Oct 30 17:49:36.359169 ip-10-0-128-236 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 17:49:36.359180 ip-10-0-128-236 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 17:49:36.359188 ip-10-0-128-236 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 17:49:36.359490 ip-10-0-128-236 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 17:49:46.591253 ip-10-0-128-236 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 17:49:46.591273 ip-10-0-128-236 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 53cf46c1e09545b3860fcf500870b47a -- Oct 30 17:51:40.222786 ip-10-0-128-236 systemd[1]: Starting Kubernetes Kubelet... Oct 30 17:51:40.660821 ip-10-0-128-236 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:51:40.660821 ip-10-0-128-236 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 17:51:40.660821 ip-10-0-128-236 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:51:40.660821 ip-10-0-128-236 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:51:40.667966 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.667889 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:40.667966 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.667962 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:40.667966 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.667966 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:40.667966 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.667969 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:40.668082 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.667971 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:40.668082 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.667974 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:40.668082 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.667977 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:40.668082 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.667980 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:40.668082 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.667982 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:40.668082 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.667985 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:40.668082 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.667988 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:40.668082 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.667991 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:40.668082 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.667993 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:40.668082 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.667996 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:40.668082 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.667999 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:40.668082 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668009 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:40.668082 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668012 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:40.668082 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668015 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:40.668082 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668017 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:40.668082 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668020 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:40.668082 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668022 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:40.668082 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668025 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:40.668082 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668027 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:40.668531 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668030 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:40.668531 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668033 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:40.668531 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668035 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:40.668531 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668037 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:40.668531 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668040 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:40.668531 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668042 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:40.668531 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668045 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:40.668531 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668048 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:40.668531 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668050 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:40.668531 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668054 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:40.668531 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668057 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:40.668531 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668059 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:40.668531 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668062 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:40.668531 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668065 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:40.668531 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668068 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:40.668531 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668071 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:40.668531 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668073 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:40.668531 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668076 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:40.668531 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668078 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:40.668531 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668081 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:40.668998 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668084 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:40.668998 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668086 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:40.668998 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668089 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:40.668998 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668092 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:40.668998 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668095 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:40.668998 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668099 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:40.668998 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668101 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:40.668998 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668104 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:40.668998 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668107 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:40.668998 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668109 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:40.668998 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668112 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:40.668998 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668114 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:40.668998 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668117 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:40.668998 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668119 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:40.668998 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668122 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:40.668998 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668124 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:40.668998 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668127 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:40.668998 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668129 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:40.668998 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668132 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:40.669430 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668135 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:40.669430 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668138 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:40.669430 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668141 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:40.669430 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668144 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:40.669430 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668146 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:40.669430 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668149 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:40.669430 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668152 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:40.669430 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668155 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:40.669430 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668157 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:40.669430 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668160 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:40.669430 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668163 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:40.669430 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668165 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:40.669430 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668168 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:40.669430 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668171 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:40.669430 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668173 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:40.669430 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668176 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:40.669430 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668178 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:40.669430 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668181 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:40.669879 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668185 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:40.669879 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668190 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:40.669879 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668193 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:40.669879 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668196 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:40.669879 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668199 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:40.669879 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668201 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:40.669879 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668204 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:40.669879 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668206 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:40.669879 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668209 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:40.669879 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668212 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:40.669879 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668216 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:40.669879 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668220 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:40.669879 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668222 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:40.669879 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668225 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:40.669879 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668228 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:40.669879 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668232 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:40.669879 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668234 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:40.669879 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668237 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:40.669879 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668240 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:40.670357 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668243 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:40.670357 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668246 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:40.670357 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668248 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:40.670357 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668251 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:40.670357 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668254 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:40.670357 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668257 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:40.670357 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.668260 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:40.670357 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669018 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:40.670357 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669026 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:40.670357 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669031 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:40.670357 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669034 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:40.670357 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669038 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:40.670357 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669041 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:40.670357 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669044 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:40.670357 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669047 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:40.670357 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669050 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:40.670357 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669053 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:40.670357 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669056 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:40.670357 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669058 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:40.670357 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669061 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:40.670853 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669063 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:40.670853 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669066 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:40.670853 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669069 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:40.670853 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669072 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:40.670853 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669075 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:40.670853 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669077 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:40.670853 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669080 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:40.670853 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669082 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:40.670853 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669086 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:40.670853 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669088 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:40.670853 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669091 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:40.670853 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669094 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:40.670853 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669097 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:40.670853 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669099 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:40.670853 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669102 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:40.670853 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669105 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:40.670853 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669107 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:40.670853 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669110 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:40.670853 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669112 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:40.671304 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669115 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:40.671304 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669118 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:40.671304 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669120 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:40.671304 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669123 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:40.671304 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669125 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:40.671304 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669128 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:40.671304 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669131 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:40.671304 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669133 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:40.671304 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669136 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:40.671304 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669139 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:40.671304 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669142 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:40.671304 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669144 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:40.671304 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669147 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:40.671304 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669149 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:40.671304 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669152 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:40.671304 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669154 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:40.671304 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669157 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:40.671304 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669159 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:40.671304 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669162 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:40.671760 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669164 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:40.671760 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669167 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:40.671760 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669170 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:40.671760 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669172 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:40.671760 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669175 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:40.671760 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669177 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:40.671760 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669181 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:40.671760 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669184 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:40.671760 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669186 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:40.671760 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669189 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:40.671760 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669191 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:40.671760 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669194 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:40.671760 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669196 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:40.671760 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669199 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:40.671760 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669201 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:40.671760 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669204 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:40.671760 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669207 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:40.671760 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669209 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:40.671760 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669212 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:40.671760 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669215 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:40.672229 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669217 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:40.672229 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669220 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:40.672229 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669223 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:40.672229 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669226 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:40.672229 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669228 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:40.672229 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669231 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:40.672229 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669233 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:40.672229 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669236 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:40.672229 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669239 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:40.672229 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669241 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:40.672229 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669244 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:40.672229 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669246 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:40.672229 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669249 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:40.672229 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669252 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:40.672229 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669258 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:40.672229 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669261 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:40.672229 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669264 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:40.672229 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669267 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:40.672229 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669270 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:40.672695 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669273 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:40.672695 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669276 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:40.672695 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669278 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:40.672695 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669281 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:40.672695 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669284 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:40.672695 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669288 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:40.672695 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669292 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:40.672695 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669295 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:40.672695 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669298 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:40.672695 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669300 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:40.672695 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669303 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:40.672695 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669305 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:40.672695 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669308 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:40.672695 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669311 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:40.672695 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669313 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:40.672695 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669316 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:40.672695 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669793 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:40.672695 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669798 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:40.672695 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669801 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:40.672695 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669804 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:40.673166 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669807 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:40.673166 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669810 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:40.673166 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669813 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:40.673166 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669816 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:40.673166 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669819 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:40.673166 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669823 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:40.673166 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669827 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:40.673166 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669830 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:40.673166 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669833 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:40.673166 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669835 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:40.673166 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669838 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:40.673166 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669841 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:40.673166 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669844 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:40.673166 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669847 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:40.673166 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669849 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:40.673166 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669852 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:40.673166 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669855 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:40.673166 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669857 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:40.673166 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669860 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:40.673166 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669863 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:40.673644 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669865 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:40.673644 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669868 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:40.673644 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669870 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:40.673644 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669873 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:40.673644 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669876 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:40.673644 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669878 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:40.673644 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669881 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:40.673644 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669883 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:40.673644 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669886 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:40.673644 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669900 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:40.673644 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669904 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:40.673644 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669907 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:40.673644 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669910 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:40.673644 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669912 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:40.673644 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669915 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:40.673644 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669918 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:40.673644 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669921 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:40.673644 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669923 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:40.673644 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669926 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:40.674090 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669928 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:40.674090 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669931 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:40.674090 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669934 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:40.674090 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669936 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:40.674090 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669939 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:40.674090 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669942 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:40.674090 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669944 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:40.674090 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669947 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:40.674090 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669950 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:40.674090 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669953 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:40.674090 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669955 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:40.674090 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669958 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:40.674090 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669961 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:40.674090 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669964 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:40.674090 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669966 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:40.674090 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669969 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:40.674090 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669971 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:40.674090 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669974 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:40.674090 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669976 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:40.674580 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669979 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:40.674580 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669981 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:40.674580 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669984 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:40.674580 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669987 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:40.674580 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669990 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:40.674580 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669993 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:40.674580 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669995 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:40.674580 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.669998 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:40.674580 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670001 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:40.674580 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670003 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:40.674580 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670006 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:40.674580 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670008 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:40.674580 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670011 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:40.674580 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670014 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:40.674580 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670016 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:40.674580 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670020 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:40.674580 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670022 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:40.674580 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670025 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:40.674580 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670028 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:40.675031 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670031 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:40.675031 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670034 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:40.675031 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670036 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:40.675031 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670039 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:40.675031 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670042 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:40.675031 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670045 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:40.675031 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670047 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:40.675031 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670050 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:40.675031 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670053 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:40.675031 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670057 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:40.675031 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670060 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:40.675031 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670063 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:40.675031 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670066 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:40.675031 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670068 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:40.675031 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670071 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:40.675031 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670073 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:40.675031 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670076 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:40.675031 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670078 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:40.675031 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670081 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:40.675031 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670084 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:40.675508 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670087 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:40.675508 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670090 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:40.675508 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670093 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:40.675508 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670095 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:40.675508 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.670098 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:40.675508 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671168 2578 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 17:51:40.675508 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671177 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 17:51:40.675508 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671187 2578 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 17:51:40.675508 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671192 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 17:51:40.675508 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671196 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 17:51:40.675508 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671200 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 17:51:40.675508 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671205 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 17:51:40.675508 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671209 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 17:51:40.675508 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671212 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 17:51:40.675508 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671215 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 17:51:40.675508 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671219 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 17:51:40.675508 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671222 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 17:51:40.675508 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671226 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 17:51:40.675508 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671229 2578 flags.go:64] FLAG: --cgroup-root="" Oct 30 17:51:40.675508 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671232 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 17:51:40.675508 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671235 2578 flags.go:64] FLAG: --client-ca-file="" Oct 30 17:51:40.675508 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671238 2578 flags.go:64] FLAG: --cloud-provider="external" Oct 30 17:51:40.675508 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671241 2578 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671245 2578 flags.go:64] FLAG: --cluster-domain="" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671248 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671251 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671254 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671257 2578 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671261 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671264 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671267 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671273 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671276 2578 flags.go:64] FLAG: --contention-profiling="false" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671279 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671282 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671286 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671289 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671293 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671296 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671299 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671302 2578 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671305 2578 flags.go:64] FLAG: --enable-server="true" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671308 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671313 2578 flags.go:64] FLAG: --event-burst="100" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671316 2578 flags.go:64] FLAG: --event-qps="50" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671319 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 17:51:40.676064 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671323 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671326 2578 flags.go:64] FLAG: --eviction-hard="" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671330 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671333 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671336 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671339 2578 flags.go:64] FLAG: --eviction-soft="" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671342 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671345 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671348 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671351 2578 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671354 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671357 2578 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671360 2578 flags.go:64] FLAG: --feature-gates="" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671364 2578 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671367 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671370 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671373 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671378 2578 flags.go:64] FLAG: --healthz-port="10248" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671381 2578 flags.go:64] FLAG: --help="false" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671384 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-128-236.ec2.internal" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671388 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671391 2578 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671394 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671397 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 17:51:40.676648 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671401 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671403 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671406 2578 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671409 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671412 2578 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671415 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671418 2578 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671421 2578 flags.go:64] FLAG: --kube-reserved="" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671424 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671427 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671431 2578 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671434 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671437 2578 flags.go:64] FLAG: --lock-file="" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671439 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671442 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671445 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671451 2578 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671454 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671457 2578 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671460 2578 flags.go:64] FLAG: --logging-format="text" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671463 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671467 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671470 2578 flags.go:64] FLAG: --manifest-url="" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671473 2578 flags.go:64] FLAG: --manifest-url-header="" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671477 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 17:51:40.677211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671481 2578 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671485 2578 flags.go:64] FLAG: --max-pods="110" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671488 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671491 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671494 2578 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671497 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671512 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671515 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671518 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671527 2578 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671530 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671533 2578 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671536 2578 flags.go:64] FLAG: --pod-cidr="" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671540 2578 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671543 2578 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671546 2578 flags.go:64] FLAG: --pods-per-core="0" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671549 2578 flags.go:64] FLAG: --port="10250" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671552 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671555 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c1b882d1b325481e" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671558 2578 flags.go:64] FLAG: --qos-reserved="" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671561 2578 flags.go:64] FLAG: --read-only-port="10255" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671564 2578 flags.go:64] FLAG: --register-node="true" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671567 2578 flags.go:64] FLAG: --register-with-taints="" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671571 2578 flags.go:64] FLAG: --registry-burst="10" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671574 2578 flags.go:64] FLAG: --registry-qps="5" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671577 2578 flags.go:64] FLAG: --reserved-cpus="" Oct 30 17:51:40.677959 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671579 2578 flags.go:64] FLAG: --reserved-memory="" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671583 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671586 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671589 2578 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671592 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671595 2578 flags.go:64] FLAG: --runonce="false" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671599 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671602 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671605 2578 flags.go:64] FLAG: --seccomp-default="false" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671609 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671612 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671615 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671618 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671621 2578 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671624 2578 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671628 2578 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671632 2578 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671635 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671638 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671641 2578 flags.go:64] FLAG: --system-cgroups="" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671644 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671649 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671651 2578 flags.go:64] FLAG: --tls-cert-file="" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671654 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671658 2578 flags.go:64] FLAG: --tls-min-version="" Oct 30 17:51:40.679115 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671661 2578 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 17:51:40.680350 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671664 2578 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 17:51:40.680350 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671667 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 17:51:40.680350 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671670 2578 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 17:51:40.680350 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671673 2578 flags.go:64] FLAG: --v="2" Oct 30 17:51:40.680350 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671677 2578 flags.go:64] FLAG: --version="false" Oct 30 17:51:40.680350 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671681 2578 flags.go:64] FLAG: --vmodule="" Oct 30 17:51:40.680350 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671685 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 17:51:40.680350 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.671688 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 17:51:40.680350 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671837 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:40.680350 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671841 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:40.680350 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671844 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:40.680350 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671848 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:40.680350 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671853 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:40.680350 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671856 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:40.680350 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671859 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:40.680350 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671862 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:40.680350 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671865 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:40.680350 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671868 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:40.680350 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671871 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:40.680350 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671874 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:40.680350 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671876 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:40.681280 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671881 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:40.681280 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671883 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:40.681280 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671886 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:40.681280 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671889 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:40.681280 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671892 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:40.681280 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671894 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:40.681280 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671897 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:40.681280 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671899 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:40.681280 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671902 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:40.681280 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671905 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:40.681280 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671907 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:40.681280 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671911 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:40.681280 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671913 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:40.681280 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671916 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:40.681280 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671918 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:40.681280 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671921 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:40.681280 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671923 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:40.681280 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671926 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:40.681280 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671929 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:40.682120 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671932 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:40.682120 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671934 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:40.682120 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671937 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:40.682120 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671939 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:40.682120 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671943 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:40.682120 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671946 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:40.682120 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671949 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:40.682120 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671951 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:40.682120 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671954 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:40.682120 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671957 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:40.682120 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671960 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:40.682120 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671962 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:40.682120 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671965 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:40.682120 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671968 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:40.682120 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671971 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:40.682120 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671974 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:40.682120 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671976 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:40.682120 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671979 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:40.682120 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671982 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:40.682664 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671984 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:40.682664 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671987 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:40.682664 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671990 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:40.682664 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671992 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:40.682664 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671995 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:40.682664 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.671998 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:40.682664 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672000 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:40.682664 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672003 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:40.682664 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672005 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:40.682664 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672008 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:40.682664 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672010 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:40.682664 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672014 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:40.682664 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672018 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:40.682664 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672021 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:40.682664 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672024 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:40.682664 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672026 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:40.682664 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672030 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:40.682664 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672035 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:40.682664 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672038 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:40.683454 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672041 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:40.683454 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672044 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:40.683454 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672046 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:40.683454 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672049 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:40.683454 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672052 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:40.683454 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672055 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:40.683454 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672058 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:40.683454 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672062 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:40.683454 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672064 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:40.683454 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672067 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:40.683454 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672070 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:40.683454 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672072 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:40.683454 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672075 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:40.683454 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672077 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:40.683454 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672080 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:40.683454 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672082 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:40.683454 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672085 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:40.683454 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672087 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:40.683454 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672090 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:40.683454 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672092 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:40.684108 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672095 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:40.684108 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672098 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:40.684108 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672100 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:40.684108 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672103 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:40.684108 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672106 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:40.684108 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672108 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:40.684108 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672111 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:40.684108 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672113 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:40.684108 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672116 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:40.684108 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672118 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:40.684108 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672122 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:40.684108 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672124 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:40.684108 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672127 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:40.684108 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672129 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:40.684108 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672132 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:40.684108 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672134 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:40.684708 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.672160 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:51:40.684708 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672195 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:40.684708 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672199 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:40.684708 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672201 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:40.684708 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672205 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:40.684708 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672209 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:40.684708 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672211 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:40.684708 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672214 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:40.684708 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672216 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:40.684708 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672219 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:40.684708 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672221 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:40.684708 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672224 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:40.684708 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672227 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:40.684708 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672229 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:40.684708 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672232 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:40.684708 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672234 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:40.685096 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672237 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:40.685096 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672240 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:40.685096 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672243 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:40.685096 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672246 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:40.685096 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672248 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:40.685096 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672251 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:40.685096 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672254 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:40.685096 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672256 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:40.685096 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672259 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:40.685096 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672261 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:40.685096 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672264 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:40.685096 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672267 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:40.685096 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672270 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:40.685096 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672272 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:40.685096 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672275 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:40.685096 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672277 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:40.685096 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672280 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:40.685096 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672282 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:40.685096 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672285 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:40.685096 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672288 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:40.685729 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672291 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:40.685729 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672294 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:40.685729 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672296 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:40.685729 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672299 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:40.685729 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672302 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:40.685729 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672304 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:40.685729 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672307 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:40.685729 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672309 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:40.685729 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672312 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:40.685729 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672314 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:40.685729 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672317 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:40.685729 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672319 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:40.685729 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672323 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:40.685729 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672325 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:40.685729 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672328 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:40.685729 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672330 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:40.685729 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672333 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:40.685729 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672335 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:40.685729 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672338 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:40.685729 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672340 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:40.686194 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672343 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:40.686194 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672345 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:40.686194 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672348 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:40.686194 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672351 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:40.686194 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672353 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:40.686194 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672356 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:40.686194 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672358 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:40.686194 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672361 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:40.686194 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672364 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:40.686194 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672366 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:40.686194 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672369 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:40.686194 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672372 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:40.686194 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672375 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:40.686194 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672377 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:40.686194 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672380 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:40.686194 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672382 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:40.686194 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672384 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:40.686194 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672387 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:40.686194 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672389 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:40.686797 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672392 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:40.686797 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672394 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:40.686797 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672397 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:40.686797 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672399 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:40.686797 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672402 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:40.686797 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672404 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:40.686797 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672407 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:40.686797 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672409 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:40.686797 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672412 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:40.686797 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672414 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:40.686797 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672417 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:40.686797 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672419 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:40.686797 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672422 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:40.686797 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672424 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:40.686797 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672427 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:40.686797 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672429 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:40.686797 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672432 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:40.686797 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672434 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:40.686797 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672437 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:40.687251 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672441 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:40.687251 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672445 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:40.687251 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672448 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:40.687251 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672451 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:40.687251 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672453 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:40.687251 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672458 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:40.687251 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672462 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:40.687251 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672465 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:40.687251 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672467 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:40.687251 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672470 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:40.687251 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672473 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:40.687251 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672475 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:40.687251 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.672478 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:40.687251 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.679997 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 17:51:40.687251 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.680017 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 17:51:40.687251 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.680037 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 17:51:40.687251 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.680044 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 17:51:40.687251 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680124 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:40.687251 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680132 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:40.687751 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680138 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:40.687751 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680143 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:40.687751 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680148 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:40.687751 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680152 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:40.687751 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680157 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:40.687751 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680162 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:40.687751 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680167 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:40.687751 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680172 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:40.687751 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680176 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:40.687751 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680181 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:40.687751 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680185 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:40.687751 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680190 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:40.687751 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680194 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:40.687751 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680199 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:40.687751 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680203 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:40.687751 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680207 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:40.687751 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680212 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:40.687751 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680216 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:40.687751 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680220 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:40.687751 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680224 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:40.688222 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680229 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:40.688222 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680233 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:40.688222 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680238 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:40.688222 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680242 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:40.688222 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680246 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:40.688222 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680251 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:40.688222 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680256 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:40.688222 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680262 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:40.688222 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680266 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:40.688222 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680271 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:40.688222 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680276 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:40.688222 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680280 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:40.688222 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680285 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:40.688222 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680289 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:40.688222 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680293 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:40.688222 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680298 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:40.688222 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680302 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:40.688222 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680306 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:40.688222 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680310 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:40.688970 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680315 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:40.688970 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680319 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:40.688970 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680324 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:40.688970 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680329 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:40.688970 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680333 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:40.688970 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680337 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:40.688970 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680342 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:40.688970 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680346 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:40.688970 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680350 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:40.688970 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680354 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:40.688970 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680358 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:40.688970 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680362 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:40.688970 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680366 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:40.688970 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680370 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:40.688970 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680374 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:40.688970 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680378 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:40.688970 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680382 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:40.688970 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680387 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:40.688970 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680391 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:40.689719 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680396 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:40.689719 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680401 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:40.689719 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680406 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:40.689719 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680411 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:40.689719 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680416 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:40.689719 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680420 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:40.689719 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680424 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:40.689719 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680428 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:40.689719 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680433 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:40.689719 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680437 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:40.689719 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680441 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:40.689719 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680448 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:40.689719 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680455 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:40.689719 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680460 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:40.689719 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680465 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:40.689719 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680470 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:40.689719 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680475 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:40.689719 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680479 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:40.689719 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680484 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:40.690203 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680488 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:40.690203 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680493 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:40.690203 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680497 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:40.690203 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680523 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:40.690203 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680528 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:40.690203 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680532 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:40.690203 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680536 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:40.690203 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680541 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:40.690203 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680545 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:40.690203 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680549 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:40.690203 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680554 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:40.690203 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680558 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:40.690203 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680562 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:40.690203 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680567 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:40.690203 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680572 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:40.690203 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680576 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:40.690203 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680582 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:40.690203 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680586 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:40.690203 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680592 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:40.690203 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680597 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:40.690745 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680603 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:40.690745 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680607 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:40.690745 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680612 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:40.690745 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680616 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:40.690745 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680621 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:40.690745 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680626 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:40.690745 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680630 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:40.690745 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.680673 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:51:40.690745 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680910 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:40.690745 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680919 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:40.690745 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680924 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:40.690745 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680929 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:40.690745 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680934 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:40.690745 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680938 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:40.690745 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680942 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:40.690745 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680947 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:40.691196 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680952 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:40.691196 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680957 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:40.691196 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680962 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:40.691196 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680966 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:40.691196 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680970 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:40.691196 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680975 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:40.691196 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680979 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:40.691196 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680983 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:40.691196 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680987 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:40.691196 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680993 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:40.691196 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.680997 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:40.691196 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681002 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:40.691196 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681006 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:40.691196 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681011 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:40.691196 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681015 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:40.691196 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681019 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:40.691196 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681024 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:40.691196 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681028 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:40.691196 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681033 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:40.691659 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681037 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:40.691659 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681041 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:40.691659 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681046 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:40.691659 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681050 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:40.691659 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681055 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:40.691659 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681059 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:40.691659 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681063 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:40.691659 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681068 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:40.691659 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681073 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:40.691659 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681078 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:40.691659 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681082 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:40.691659 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681086 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:40.691659 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681090 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:40.691659 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681095 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:40.691659 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681099 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:40.691659 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681103 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:40.691659 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681107 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:40.691659 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681114 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:40.691659 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681120 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:40.692319 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681125 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:40.692319 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681129 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:40.692319 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681134 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:40.692319 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681139 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:40.692319 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681143 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:40.692319 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681148 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:40.692319 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681152 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:40.692319 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681156 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:40.692319 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681161 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:40.692319 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681166 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:40.692319 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681170 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:40.692319 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681174 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:40.692319 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681179 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:40.692319 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681183 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:40.692319 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681187 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:40.692319 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681191 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:40.692319 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681195 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:40.692319 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681200 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:40.692319 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681204 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:40.692798 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681209 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:40.692798 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681213 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:40.692798 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681217 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:40.692798 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681222 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:40.692798 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681226 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:40.692798 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681230 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:40.692798 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681235 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:40.692798 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681239 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:40.692798 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681244 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:40.692798 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681248 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:40.692798 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681255 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:40.692798 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681261 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:40.692798 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681265 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:40.692798 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681270 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:40.692798 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681274 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:40.692798 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681278 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:40.692798 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681282 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:40.692798 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681286 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:40.692798 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681290 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:40.692798 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681294 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:40.693303 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681298 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:40.693303 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681302 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:40.693303 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681307 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:40.693303 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681312 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:40.693303 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681316 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:40.693303 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681320 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:40.693303 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681325 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:40.693303 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681329 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:40.693303 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681333 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:40.693303 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681338 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:40.693303 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681342 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:40.693303 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681346 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:40.693303 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681350 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:40.693303 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681355 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:40.693303 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681359 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:40.693303 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681364 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:40.693303 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681368 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:40.693303 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681373 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:40.693303 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681377 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:40.693303 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681381 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:40.693834 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681386 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:40.693834 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.681424 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:51:40.693834 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681475 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:40.693834 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681481 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:40.693834 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681486 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:40.693834 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681492 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:40.693834 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681496 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:40.693834 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681518 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:40.693834 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681523 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:40.693834 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681527 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:40.693834 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681531 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:40.693834 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681535 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:40.693834 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681540 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:40.693834 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681544 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:40.693834 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681548 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:40.693834 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681554 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:40.694220 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681558 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:40.694220 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681563 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:40.694220 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681567 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:40.694220 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681571 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:40.694220 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681575 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:40.694220 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681580 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:40.694220 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681584 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:40.694220 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681588 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:40.694220 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681592 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:40.694220 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681597 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:40.694220 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681601 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:40.694220 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681605 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:40.694220 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681610 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:40.694220 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681614 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:40.694220 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681618 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:40.694220 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681622 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:40.694220 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681626 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:40.694220 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681630 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:40.694220 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681637 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:40.694871 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681641 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:40.694871 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681646 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:40.694871 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681650 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:40.694871 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681654 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:40.694871 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681658 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:40.694871 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681663 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:40.694871 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681667 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:40.694871 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681671 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:40.694871 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681676 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:40.694871 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681680 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:40.694871 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681684 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:40.694871 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681688 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:40.694871 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681693 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:40.694871 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681697 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:40.694871 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681702 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:40.694871 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681707 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:40.694871 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681711 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:40.694871 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681715 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:40.694871 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681720 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:40.694871 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681724 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:40.695349 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681728 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:40.695349 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681733 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:40.695349 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681737 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:40.695349 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681741 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:40.695349 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681745 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:40.695349 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681750 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:40.695349 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681754 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:40.695349 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681758 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:40.695349 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681765 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:40.695349 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681772 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:40.695349 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681778 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:40.695349 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681784 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:40.695349 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681788 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:40.695349 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681793 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:40.695349 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681797 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:40.695349 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681801 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:40.695349 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681806 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:40.695349 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681810 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:40.695349 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681814 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:40.695850 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681819 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:40.695850 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681824 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:40.695850 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681828 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:40.695850 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681832 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:40.695850 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681836 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:40.695850 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681840 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:40.695850 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681845 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:40.695850 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681851 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:40.695850 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681856 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:40.695850 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681860 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:40.695850 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681864 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:40.695850 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681869 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:40.695850 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681873 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:40.695850 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681877 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:40.695850 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681881 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:40.695850 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681886 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:40.695850 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681890 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:40.695850 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681894 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:40.695850 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681899 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:40.695850 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681903 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:40.696369 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681909 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:40.696369 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681915 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:40.696369 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681920 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:40.696369 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681924 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:40.696369 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681930 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:40.696369 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681934 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:40.696369 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681939 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:40.696369 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681943 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:40.696369 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681947 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:40.696369 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681951 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:40.696369 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681956 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:40.696369 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681961 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:40.696369 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681965 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:40.696369 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:40.681969 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:40.696369 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.682047 2578 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 17:51:40.696369 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.685416 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 17:51:40.696369 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.686367 2578 server.go:1014] "Starting client certificate rotation" Oct 30 17:51:40.696369 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.686468 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:51:40.696369 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.686529 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:51:40.696878 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.688486 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:51:40.696878 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.693014 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:51:40.705317 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.705302 2578 log.go:25] "Validated CRI v1 runtime API" Oct 30 17:51:40.708346 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.708331 2578 log.go:25] "Validated CRI v1 image API" Oct 30 17:51:40.709217 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.709192 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 17:51:40.713689 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.713670 2578 fs.go:135] Filesystem UUIDs: map[0ebb35b0-35ed-47cf-8a34-7459d7fb119c:/dev/nvme0n1p3 25cf32f3-6603-46f4-9061-490321d7fddf:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 30 17:51:40.713751 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.713690 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 17:51:40.719376 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.719269 2578 manager.go:217] Machine: {Timestamp:2026-10-30 17:51:40.717873315 +0000 UTC m=+0.357206113 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097899 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ce0bb041ebbb460e8fc4e4f86656a SystemUUID:ec2ce0bb-041e-bbb4-60e8-fc4e4f86656a BootID:53cf46c1-e095-45b3-860f-cf500870b47a Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c0:49:dc:bf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c0:49:dc:bf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:bf:2c:7f:db:05 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 17:51:40.719443 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.719377 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 17:51:40.719477 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.719452 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 17:51:40.722409 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.722389 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 17:51:40.722568 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.722411 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-236.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 17:51:40.722615 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.722571 2578 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 17:51:40.722615 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.722579 2578 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 17:51:40.722615 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.722609 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:51:40.722693 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.722627 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:51:40.723433 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.723424 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 17:51:40.726282 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.726271 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 17:51:40.726400 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.726391 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 17:51:40.727113 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.727104 2578 kubelet.go:493] "Attempting to sync node with API server" Oct 30 17:51:40.727144 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.727118 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 17:51:40.727144 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.727129 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 17:51:40.727144 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.727137 2578 kubelet.go:405] "Adding apiserver pod source" Oct 30 17:51:40.727235 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.727144 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 17:51:40.727583 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.727569 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:51:40.731744 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.731723 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 17:51:40.736175 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.736161 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:51:40.736991 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.736979 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 17:51:40.737036 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.737010 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 17:51:40.737156 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.737146 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 17:51:40.737183 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.737161 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 17:51:40.737183 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.737167 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 17:51:40.737183 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.737173 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 17:51:40.737183 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.737179 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 17:51:40.737183 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.737184 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 17:51:40.737794 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.737785 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 17:51:40.737824 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.737795 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 17:51:40.737824 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.737803 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 17:51:40.737824 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.737808 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 17:51:40.737824 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.737816 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 17:51:40.737938 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.737826 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 17:51:40.738465 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.738457 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 17:51:40.738495 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.738467 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 17:51:40.742020 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.742009 2578 server.go:1263] "Started kubelet" Oct 30 17:51:40.742098 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.742068 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 17:51:40.742168 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.742121 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 17:51:40.742230 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.742218 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 17:51:40.742743 ip-10-0-128-236 systemd[1]: Started Kubernetes Kubelet. Oct 30 17:51:40.743085 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.743068 2578 server.go:317] "Adding debug handlers to kubelet server" Oct 30 17:51:40.743689 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.743437 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 17:51:40.748890 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:40.748865 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 17:51:40.749701 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.749683 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 17:51:40.750730 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.750714 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 17:51:40.750866 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:40.750847 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-236.ec2.internal\" not found" Oct 30 17:51:40.751435 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.751417 2578 factory.go:55] Registering systemd factory Oct 30 17:51:40.751435 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.751421 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 17:51:40.751556 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.751444 2578 factory.go:223] Registration of the systemd container factory successfully Oct 30 17:51:40.751556 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.751450 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 17:51:40.751556 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.751420 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 17:51:40.751556 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.751552 2578 reconstruct.go:97] "Volume reconstruction finished" Oct 30 17:51:40.751695 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.751561 2578 reconciler.go:29] "Reconciler: start to sync state" Oct 30 17:51:40.751695 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.751672 2578 factory.go:153] Registering CRI-O factory Oct 30 17:51:40.751695 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.751685 2578 factory.go:223] Registration of the crio container factory successfully Oct 30 17:51:40.751809 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.751733 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 17:51:40.751809 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.751751 2578 factory.go:103] Registering Raw factory Oct 30 17:51:40.751809 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.751760 2578 manager.go:1199] Started watching for new ooms in manager Oct 30 17:51:40.753014 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.752997 2578 manager.go:319] Starting recovery of all containers Oct 30 17:51:40.765614 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.765599 2578 manager.go:324] Recovery completed Oct 30 17:51:40.769424 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.769410 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:51:40.772007 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.771990 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:51:40.772072 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.772018 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:51:40.772072 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.772032 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:51:40.772711 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.772698 2578 cpu_manager.go:226] "Starting" policy="none" Oct 30 17:51:40.772711 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.772708 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 17:51:40.772822 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.772723 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 17:51:40.772822 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.772769 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:51:40.773620 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.773516 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:51:40.773677 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.773633 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:51:40.773677 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.773644 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:51:40.773865 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.773854 2578 policy_none.go:50] "Start" Oct 30 17:51:40.773901 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.773869 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 17:51:40.773901 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.773878 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 17:51:40.774824 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.774814 2578 policy_none.go:44] "Start" Oct 30 17:51:40.786309 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.786290 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 17:51:40.786409 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.786351 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5kn4l" Oct 30 17:51:40.792804 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.792786 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-236.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:51:40.792886 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:40.792850 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 17:51:40.793038 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:40.793015 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 17:51:40.793924 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:40.792750 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-236.ec2.internal.18e36010880e7d9a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-236.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-236.ec2.internal,},FirstTimestamp:2026-10-30 17:51:40.741983642 +0000 UTC m=+0.381316441,LastTimestamp:2026-10-30 17:51:40.741983642 +0000 UTC m=+0.381316441,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-236.ec2.internal,}" Oct 30 17:51:40.795702 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.795678 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5kn4l" Oct 30 17:51:40.827014 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.824860 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 17:51:40.827014 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.825454 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 17:51:40.827014 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.826794 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 17:51:40.827014 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.826818 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 17:51:40.827014 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.826838 2578 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 17:51:40.827014 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:40.826876 2578 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 17:51:40.827014 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.826990 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:51:40.827674 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.827657 2578 manager.go:341] "Starting Device Plugin manager" Oct 30 17:51:40.827770 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:40.827694 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 17:51:40.827770 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.827706 2578 server.go:87] "Starting device plugin registration server" Oct 30 17:51:40.827978 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.827966 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 17:51:40.828054 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.827979 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 17:51:40.828180 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.828163 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 17:51:40.828251 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.828239 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 17:51:40.828320 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.828252 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 17:51:40.829588 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:40.829566 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 17:51:40.829686 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:40.829657 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-236.ec2.internal\" not found" Oct 30 17:51:40.850393 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.850377 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:51:40.851417 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.851402 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:51:40.927596 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.927541 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal"] Oct 30 17:51:40.927596 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.927596 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:51:40.928231 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.928213 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:51:40.928565 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.928548 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:51:40.928650 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.928581 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:51:40.928650 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.928597 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:51:40.928998 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.928983 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:51:40.929051 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.929010 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:51:40.929051 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.929024 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:51:40.929051 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.929041 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-236.ec2.internal" Oct 30 17:51:40.930083 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.930069 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:51:40.930235 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.930221 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" Oct 30 17:51:40.930277 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.930249 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:51:40.930808 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.930784 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:51:40.930913 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.930815 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:51:40.930913 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.930862 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:51:40.930913 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.930785 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:51:40.930913 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.930894 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:51:40.930913 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.930906 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:51:40.931914 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.931899 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal" Oct 30 17:51:40.931996 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.931935 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:51:40.932607 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.932590 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:51:40.932700 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.932614 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:51:40.932700 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.932623 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:51:40.947409 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.947390 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-236.ec2.internal" Oct 30 17:51:40.947637 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.947614 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" Oct 30 17:51:40.951022 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.951007 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal" Oct 30 17:51:40.951096 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.951087 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" Oct 30 17:51:40.953258 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.953239 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb76617dbf2b01fcc9f68e4666f49e08-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal\" (UID: \"bb76617dbf2b01fcc9f68e4666f49e08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" Oct 30 17:51:40.953354 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.953263 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb76617dbf2b01fcc9f68e4666f49e08-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal\" (UID: \"bb76617dbf2b01fcc9f68e4666f49e08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" Oct 30 17:51:40.953354 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.953278 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f50f91e38dadbc11b70b476e358bc37f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-236.ec2.internal\" (UID: \"f50f91e38dadbc11b70b476e358bc37f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal" Oct 30 17:51:40.976754 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.976734 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:51:40.976840 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.976759 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:51:40.980550 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.980531 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:51:40.980656 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:40.980585 2578 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" Oct 30 17:51:40.980656 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:40.980600 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal" Oct 30 17:51:41.007566 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.007551 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:51:41.007661 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:41.007584 2578 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-236.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal" Oct 30 17:51:41.053661 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.053641 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb76617dbf2b01fcc9f68e4666f49e08-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal\" (UID: \"bb76617dbf2b01fcc9f68e4666f49e08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" Oct 30 17:51:41.053722 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.053678 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f50f91e38dadbc11b70b476e358bc37f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-236.ec2.internal\" (UID: \"f50f91e38dadbc11b70b476e358bc37f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal" Oct 30 17:51:41.053722 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.053701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb76617dbf2b01fcc9f68e4666f49e08-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal\" (UID: \"bb76617dbf2b01fcc9f68e4666f49e08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" Oct 30 17:51:41.053807 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.053716 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb76617dbf2b01fcc9f68e4666f49e08-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal\" (UID: \"bb76617dbf2b01fcc9f68e4666f49e08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" Oct 30 17:51:41.053807 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.053740 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb76617dbf2b01fcc9f68e4666f49e08-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal\" (UID: \"bb76617dbf2b01fcc9f68e4666f49e08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" Oct 30 17:51:41.053807 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.053748 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f50f91e38dadbc11b70b476e358bc37f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-236.ec2.internal\" (UID: \"f50f91e38dadbc11b70b476e358bc37f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal" Oct 30 17:51:41.283125 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.283060 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" Oct 30 17:51:41.283643 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.283624 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal" Oct 30 17:51:41.680237 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.680211 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:51:41.686291 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.686267 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 17:51:41.686451 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.686423 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 17:51:41.686535 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.686485 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 17:51:41.686582 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.686548 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 17:51:41.718896 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:41.718870 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf50f91e38dadbc11b70b476e358bc37f.slice/crio-81c432afc2475543bfeda559a25113dc69a47425c99a1a739b7416bf9acf983d WatchSource:0}: Error finding container 81c432afc2475543bfeda559a25113dc69a47425c99a1a739b7416bf9acf983d: Status 404 returned error can't find the container with id 81c432afc2475543bfeda559a25113dc69a47425c99a1a739b7416bf9acf983d Oct 30 17:51:41.720789 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:41.720773 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb76617dbf2b01fcc9f68e4666f49e08.slice/crio-bd5a6a5bef5bd27b9d1982aabd2157ac61ec13df9888f62fb9027c666fcfbe0e WatchSource:0}: Error finding container bd5a6a5bef5bd27b9d1982aabd2157ac61ec13df9888f62fb9027c666fcfbe0e: Status 404 returned error can't find the container with id bd5a6a5bef5bd27b9d1982aabd2157ac61ec13df9888f62fb9027c666fcfbe0e Oct 30 17:51:41.724406 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.724390 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:51:41.727485 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.727470 2578 apiserver.go:52] "Watching apiserver" Oct 30 17:51:41.750701 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.750678 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 17:51:41.759858 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.759842 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 17:51:41.760109 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.760095 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-th8wr","openshift-network-diagnostics/network-check-target-4xxmr","openshift-network-operator/iptables-alerter-db4lm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal","openshift-multus/multus-additional-cni-plugins-v5l7g","openshift-multus/network-metrics-daemon-jnxk7","openshift-ovn-kubernetes/ovnkube-node-7cgxg","kube-system/global-pull-secret-syncer-5drgz","kube-system/konnectivity-agent-pgghg","kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal","openshift-cluster-node-tuning-operator/tuned-khz68","openshift-image-registry/node-ca-4hx5j","openshift-dns/node-resolver-pbqhh"] Oct 30 17:51:41.761841 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.761825 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.762721 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.762707 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:51:41.762808 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:41.762781 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxmr" podUID="90437f91-1657-47db-b3a1-d59bc8d1de14" Oct 30 17:51:41.763710 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.763697 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-db4lm" Oct 30 17:51:41.764716 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.764698 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:51:41.765762 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.765740 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:51:41.766682 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.766666 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:51:41.766779 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:41.766759 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnxk7" podUID="827bbba0-b306-43db-b805-8c87dfabb874" Oct 30 17:51:41.767859 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.767843 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.768944 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.768930 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:51:41.768999 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:41.768977 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5drgz" podUID="1e6590e8-bfc4-4b19-b579-7180aa6af22a" Oct 30 17:51:41.770004 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.769990 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pgghg" Oct 30 17:51:41.771076 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.771057 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.772310 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.772294 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4hx5j" Oct 30 17:51:41.773374 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.773359 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pbqhh" Oct 30 17:51:41.774360 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.774332 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 17:51:41.774360 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.774344 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 17:51:41.774360 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.774341 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 17:51:41.774545 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.774379 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 17:51:41.774545 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.774399 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dqbz4\"" Oct 30 17:51:41.776424 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.776409 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hhs6k\"" Oct 30 17:51:41.796522 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.796489 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 17:51:41.796589 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.796559 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-w9djw" Oct 30 17:51:41.799406 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.799361 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:46:41 +0000 UTC" deadline="2028-07-07 12:10:56.684365238 +0000 UTC" Oct 30 17:51:41.799484 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.799407 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14778h19m14.884961853s" Oct 30 17:51:41.809258 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.809244 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-w9djw" Oct 30 17:51:41.831660 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.831604 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" event={"ID":"bb76617dbf2b01fcc9f68e4666f49e08","Type":"ContainerStarted","Data":"bd5a6a5bef5bd27b9d1982aabd2157ac61ec13df9888f62fb9027c666fcfbe0e"} Oct 30 17:51:41.832707 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.832686 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal" event={"ID":"f50f91e38dadbc11b70b476e358bc37f","Type":"ContainerStarted","Data":"81c432afc2475543bfeda559a25113dc69a47425c99a1a739b7416bf9acf983d"} Oct 30 17:51:41.853069 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.853051 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 17:51:41.859267 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859252 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f-konnectivity-ca\") pod \"konnectivity-agent-pgghg\" (UID: \"1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f\") " pod="kube-system/konnectivity-agent-pgghg" Oct 30 17:51:41.859331 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859275 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/088d4d32-c02b-4da2-9435-c5bc22352079-host\") pod \"node-ca-4hx5j\" (UID: \"088d4d32-c02b-4da2-9435-c5bc22352079\") " pod="openshift-image-registry/node-ca-4hx5j" Oct 30 17:51:41.859331 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859290 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef04039-ef85-40ef-8f30-a0edd7b402e0-tmp-dir\") pod \"node-resolver-pbqhh\" (UID: \"3ef04039-ef85-40ef-8f30-a0edd7b402e0\") " pod="openshift-dns/node-resolver-pbqhh" Oct 30 17:51:41.859331 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859306 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-hostroot\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.859331 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859320 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fd6cf358-80f3-42f6-9848-6386a937a57a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:51:41.859484 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859364 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e9933e90-d394-4e89-8e0f-115298c359fa-iptables-alerter-script\") pod \"iptables-alerter-db4lm\" (UID: \"e9933e90-d394-4e89-8e0f-115298c359fa\") " pod="openshift-network-operator/iptables-alerter-db4lm" Oct 30 17:51:41.859484 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859388 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-socket-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:51:41.859484 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859404 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-device-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:51:41.859484 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859419 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3ef04039-ef85-40ef-8f30-a0edd7b402e0-hosts-file\") pod \"node-resolver-pbqhh\" (UID: \"3ef04039-ef85-40ef-8f30-a0edd7b402e0\") " pod="openshift-dns/node-resolver-pbqhh" Oct 30 17:51:41.859623 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859527 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-cni-binary-copy\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.859623 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859543 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-run-k8s-cni-cncf-io\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.859623 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859563 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-conf-dir\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.859623 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859598 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbs6z\" (UniqueName: \"kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z\") pod \"network-check-target-4xxmr\" (UID: \"90437f91-1657-47db-b3a1-d59bc8d1de14\") " pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:51:41.859739 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859643 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-run\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.859739 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859660 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-var-lib-kubelet\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.859739 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859678 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-var-lib-cni-multus\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.859739 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859707 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-var-lib-kubelet\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.859739 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859728 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-kubelet\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.859925 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859750 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-run-ovn-kubernetes\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.859925 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859772 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-host\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.859925 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859817 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-tuned\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.859925 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859850 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-sys-fs\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:51:41.859925 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859873 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/088d4d32-c02b-4da2-9435-c5bc22352079-serviceca\") pod \"node-ca-4hx5j\" (UID: \"088d4d32-c02b-4da2-9435-c5bc22352079\") " pod="openshift-image-registry/node-ca-4hx5j" Oct 30 17:51:41.859925 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859902 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-cni-dir\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.860093 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859938 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/95709b19-3c85-4ef1-9110-c23a97847732-env-overrides\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.860093 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859963 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t2rp\" (UniqueName: \"kubernetes.io/projected/088d4d32-c02b-4da2-9435-c5bc22352079-kube-api-access-4t2rp\") pod \"node-ca-4hx5j\" (UID: \"088d4d32-c02b-4da2-9435-c5bc22352079\") " pod="openshift-image-registry/node-ca-4hx5j" Oct 30 17:51:41.860093 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.859986 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm6vb\" (UniqueName: \"kubernetes.io/projected/3ef04039-ef85-40ef-8f30-a0edd7b402e0-kube-api-access-lm6vb\") pod \"node-resolver-pbqhh\" (UID: \"3ef04039-ef85-40ef-8f30-a0edd7b402e0\") " pod="openshift-dns/node-resolver-pbqhh" Oct 30 17:51:41.860093 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860003 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd6cf358-80f3-42f6-9848-6386a937a57a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:51:41.860093 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860026 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-run-openvswitch\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.860093 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860052 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-sysctl-conf\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.860093 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860079 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9933e90-d394-4e89-8e0f-115298c359fa-host-slash\") pod \"iptables-alerter-db4lm\" (UID: \"e9933e90-d394-4e89-8e0f-115298c359fa\") " pod="openshift-network-operator/iptables-alerter-db4lm" Oct 30 17:51:41.860289 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860101 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj9k2\" (UniqueName: \"kubernetes.io/projected/827bbba0-b306-43db-b805-8c87dfabb874-kube-api-access-nj9k2\") pod \"network-metrics-daemon-jnxk7\" (UID: \"827bbba0-b306-43db-b805-8c87dfabb874\") " pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:51:41.860289 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860118 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fd6cf358-80f3-42f6-9848-6386a937a57a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:51:41.860289 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860159 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-systemd\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.860289 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:51:41.860289 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860208 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd6cf358-80f3-42f6-9848-6386a937a57a-cnibin\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:51:41.860289 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860237 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-var-lib-openvswitch\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.860289 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860263 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-node-log\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.860289 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860290 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-daemon-config\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.860534 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860308 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-etc-kubernetes\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.860534 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860340 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd6cf358-80f3-42f6-9848-6386a937a57a-system-cni-dir\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:51:41.860534 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860367 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.860534 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860383 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-kubernetes\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.860534 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860401 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-var-lib-cni-bin\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.860534 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860416 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ks5l\" (UniqueName: \"kubernetes.io/projected/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-kube-api-access-8ks5l\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.860534 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860451 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-slash\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.860534 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860481 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/95709b19-3c85-4ef1-9110-c23a97847732-ovnkube-script-lib\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.860534 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860496 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e6590e8-bfc4-4b19-b579-7180aa6af22a-dbus\") pod \"global-pull-secret-syncer-5drgz\" (UID: \"1e6590e8-bfc4-4b19-b579-7180aa6af22a\") " pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:51:41.860797 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860531 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-modprobe-d\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.860797 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860565 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-sysctl-d\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.860797 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860583 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-tmp\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.860797 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860618 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-socket-dir-parent\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.860797 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860634 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-cni-bin\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.860797 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860662 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret\") pod \"global-pull-secret-syncer-5drgz\" (UID: \"1e6590e8-bfc4-4b19-b579-7180aa6af22a\") " pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:51:41.860797 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860690 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs\") pod \"network-metrics-daemon-jnxk7\" (UID: \"827bbba0-b306-43db-b805-8c87dfabb874\") " pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:51:41.860797 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860711 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-run-netns\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.860797 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860735 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-systemd-units\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.860797 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860750 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-run-ovn\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.860797 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860776 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-lib-modules\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.861171 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860804 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-system-cni-dir\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.861171 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860835 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-etc-openvswitch\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.861171 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860854 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhlzv\" (UniqueName: \"kubernetes.io/projected/95709b19-3c85-4ef1-9110-c23a97847732-kube-api-access-dhlzv\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.861171 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860888 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e6590e8-bfc4-4b19-b579-7180aa6af22a-kubelet-config\") pod \"global-pull-secret-syncer-5drgz\" (UID: \"1e6590e8-bfc4-4b19-b579-7180aa6af22a\") " pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:51:41.861171 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860916 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f-agent-certs\") pod \"konnectivity-agent-pgghg\" (UID: \"1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f\") " pod="kube-system/konnectivity-agent-pgghg" Oct 30 17:51:41.861171 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860938 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzf62\" (UniqueName: \"kubernetes.io/projected/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-kube-api-access-wzf62\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.861171 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860965 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-cnibin\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.861171 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.860987 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fd6cf358-80f3-42f6-9848-6386a937a57a-cni-binary-copy\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:51:41.861171 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.861025 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-run-systemd\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.861171 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.861138 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-registration-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:51:41.861171 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.861168 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq24z\" (UniqueName: \"kubernetes.io/projected/7af86f87-5d83-4a8b-a0cc-88d4641406d2-kube-api-access-vq24z\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:51:41.861635 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.861206 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-log-socket\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.861635 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.861237 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-sys\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.861635 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.861258 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:51:41.861635 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.861278 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-run-multus-certs\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.861635 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.861316 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk78z\" (UniqueName: \"kubernetes.io/projected/fd6cf358-80f3-42f6-9848-6386a937a57a-kube-api-access-pk78z\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:51:41.861635 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.861370 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-run-netns\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.861635 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.861394 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/95709b19-3c85-4ef1-9110-c23a97847732-ovnkube-config\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.861635 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.861417 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-sysconfig\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.861635 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.861442 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lnnm\" (UniqueName: \"kubernetes.io/projected/e9933e90-d394-4e89-8e0f-115298c359fa-kube-api-access-2lnnm\") pod \"iptables-alerter-db4lm\" (UID: \"e9933e90-d394-4e89-8e0f-115298c359fa\") " pod="openshift-network-operator/iptables-alerter-db4lm" Oct 30 17:51:41.861635 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.861465 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-cni-netd\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.861635 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.861489 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/95709b19-3c85-4ef1-9110-c23a97847732-ovn-node-metrics-cert\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.874924 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.874904 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 17:51:41.875053 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.875040 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 17:51:41.875314 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.875301 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 17:51:41.875688 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.875623 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-k5mms\"" Oct 30 17:51:41.875688 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.875630 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 17:51:41.875688 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.875663 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 17:51:41.875688 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.875673 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:51:41.875688 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.875677 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 17:51:41.875936 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.875773 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 17:51:41.875936 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.875845 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 17:51:41.875936 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.875851 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fzft4\"" Oct 30 17:51:41.875936 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.875932 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 17:51:41.876100 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.875948 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 17:51:41.876100 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.875953 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 17:51:41.876100 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.876000 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 17:51:41.876100 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.876006 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:51:41.876100 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.876039 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 17:51:41.876100 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.876064 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 17:51:41.876513 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.876489 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 17:51:41.881289 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.881276 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 17:51:41.881343 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.881308 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 17:51:41.881343 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.881313 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k56gt\"" Oct 30 17:51:41.881447 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.881426 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 17:51:41.881492 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.881447 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 17:51:41.881492 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.881473 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lsd9l\"" Oct 30 17:51:41.881581 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.881492 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nb8hn\"" Oct 30 17:51:41.881581 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.881493 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5kfdv\"" Oct 30 17:51:41.881581 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.881528 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 17:51:41.881581 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.881547 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 17:51:41.881581 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.881565 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-twdt2\"" Oct 30 17:51:41.962716 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.962663 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t2rp\" (UniqueName: \"kubernetes.io/projected/088d4d32-c02b-4da2-9435-c5bc22352079-kube-api-access-4t2rp\") pod \"node-ca-4hx5j\" (UID: \"088d4d32-c02b-4da2-9435-c5bc22352079\") " pod="openshift-image-registry/node-ca-4hx5j" Oct 30 17:51:41.962716 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.962686 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm6vb\" (UniqueName: \"kubernetes.io/projected/3ef04039-ef85-40ef-8f30-a0edd7b402e0-kube-api-access-lm6vb\") pod \"node-resolver-pbqhh\" (UID: \"3ef04039-ef85-40ef-8f30-a0edd7b402e0\") " pod="openshift-dns/node-resolver-pbqhh" Oct 30 17:51:41.962716 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.962710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd6cf358-80f3-42f6-9848-6386a937a57a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:51:41.962904 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.962725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-run-openvswitch\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.962904 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.962739 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-sysctl-conf\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.962904 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.962754 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9933e90-d394-4e89-8e0f-115298c359fa-host-slash\") pod \"iptables-alerter-db4lm\" (UID: \"e9933e90-d394-4e89-8e0f-115298c359fa\") " pod="openshift-network-operator/iptables-alerter-db4lm" Oct 30 17:51:41.962904 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.962769 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj9k2\" (UniqueName: \"kubernetes.io/projected/827bbba0-b306-43db-b805-8c87dfabb874-kube-api-access-nj9k2\") pod \"network-metrics-daemon-jnxk7\" (UID: \"827bbba0-b306-43db-b805-8c87dfabb874\") " pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:51:41.962904 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.962800 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-run-openvswitch\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.962904 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.962824 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9933e90-d394-4e89-8e0f-115298c359fa-host-slash\") pod \"iptables-alerter-db4lm\" (UID: \"e9933e90-d394-4e89-8e0f-115298c359fa\") " pod="openshift-network-operator/iptables-alerter-db4lm" Oct 30 17:51:41.962904 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.962897 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-sysctl-conf\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.963220 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.962910 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd6cf358-80f3-42f6-9848-6386a937a57a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:51:41.963220 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963039 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fd6cf358-80f3-42f6-9848-6386a937a57a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:51:41.963220 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963068 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-systemd\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.963220 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:51:41.963220 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963099 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd6cf358-80f3-42f6-9848-6386a937a57a-cnibin\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:51:41.963220 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963120 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-var-lib-openvswitch\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.963220 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963145 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-node-log\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.963220 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-daemon-config\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.963220 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963167 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-systemd\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.963220 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963171 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd6cf358-80f3-42f6-9848-6386a937a57a-cnibin\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:51:41.963220 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963183 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-etc-kubernetes\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.963220 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963202 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-var-lib-openvswitch\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.963220 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963167 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:51:41.963829 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963229 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-node-log\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.963829 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963234 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-etc-kubernetes\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.963829 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963229 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd6cf358-80f3-42f6-9848-6386a937a57a-system-cni-dir\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:51:41.963829 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963272 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd6cf358-80f3-42f6-9848-6386a937a57a-system-cni-dir\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:51:41.963829 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963312 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.963829 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963437 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.963829 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963471 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-kubernetes\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.963829 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963523 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-var-lib-cni-bin\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.963829 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ks5l\" (UniqueName: \"kubernetes.io/projected/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-kube-api-access-8ks5l\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.963829 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963606 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-var-lib-cni-bin\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.963829 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963636 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-slash\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.963829 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963660 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fd6cf358-80f3-42f6-9848-6386a937a57a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:51:41.963829 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963681 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/95709b19-3c85-4ef1-9110-c23a97847732-ovnkube-script-lib\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.963829 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963707 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e6590e8-bfc4-4b19-b579-7180aa6af22a-dbus\") pod \"global-pull-secret-syncer-5drgz\" (UID: \"1e6590e8-bfc4-4b19-b579-7180aa6af22a\") " pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:51:41.963829 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963744 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-daemon-config\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.963829 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963756 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-slash\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.963829 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963807 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-kubernetes\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.964525 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.963852 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e6590e8-bfc4-4b19-b579-7180aa6af22a-dbus\") pod \"global-pull-secret-syncer-5drgz\" (UID: \"1e6590e8-bfc4-4b19-b579-7180aa6af22a\") " pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:51:41.964525 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.964138 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-modprobe-d\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.964525 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.964287 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-sysctl-d\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.964525 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.964381 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-tmp\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.964525 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.964420 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-modprobe-d\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.964749 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.964580 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-socket-dir-parent\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.964749 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.964621 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-cni-bin\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.964749 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.964628 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-socket-dir-parent\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.964749 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.964674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret\") pod \"global-pull-secret-syncer-5drgz\" (UID: \"1e6590e8-bfc4-4b19-b579-7180aa6af22a\") " pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:51:41.964920 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.964845 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs\") pod \"network-metrics-daemon-jnxk7\" (UID: \"827bbba0-b306-43db-b805-8c87dfabb874\") " pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:51:41.965001 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.964976 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-run-netns\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.965050 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.965013 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-systemd-units\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.965148 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.965074 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/95709b19-3c85-4ef1-9110-c23a97847732-ovnkube-script-lib\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.965148 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.965086 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-sysctl-d\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.965148 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.965105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-run-ovn\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.965148 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.965143 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-cni-bin\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.965310 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.965164 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-lib-modules\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.965310 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.965164 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-run-netns\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.965310 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.965176 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-systemd-units\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.965310 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:41.965265 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:41.965483 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.965321 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-lib-modules\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.965483 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:41.965341 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret podName:1e6590e8-bfc4-4b19-b579-7180aa6af22a nodeName:}" failed. No retries permitted until 2026-10-30 17:51:42.465311298 +0000 UTC m=+2.104644116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret") pod "global-pull-secret-syncer-5drgz" (UID: "1e6590e8-bfc4-4b19-b579-7180aa6af22a") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:41.965483 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:41.965268 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:41.965483 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.965304 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-run-ovn\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.965483 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.965300 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-system-cni-dir\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.965483 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:41.965427 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs podName:827bbba0-b306-43db-b805-8c87dfabb874 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:42.46538466 +0000 UTC m=+2.104717449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs") pod "network-metrics-daemon-jnxk7" (UID: "827bbba0-b306-43db-b805-8c87dfabb874") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:41.965772 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.965737 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 17:51:41.966594 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.966352 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-system-cni-dir\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.966594 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.966429 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-etc-openvswitch\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.966594 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.966479 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhlzv\" (UniqueName: \"kubernetes.io/projected/95709b19-3c85-4ef1-9110-c23a97847732-kube-api-access-dhlzv\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.966594 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.966527 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e6590e8-bfc4-4b19-b579-7180aa6af22a-kubelet-config\") pod \"global-pull-secret-syncer-5drgz\" (UID: \"1e6590e8-bfc4-4b19-b579-7180aa6af22a\") " pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:51:41.966594 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.966580 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f-agent-certs\") pod \"konnectivity-agent-pgghg\" (UID: \"1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f\") " pod="kube-system/konnectivity-agent-pgghg" Oct 30 17:51:41.966858 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.966611 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzf62\" (UniqueName: \"kubernetes.io/projected/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-kube-api-access-wzf62\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.966858 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.966642 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-cnibin\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.966858 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.966687 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fd6cf358-80f3-42f6-9848-6386a937a57a-cni-binary-copy\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:51:41.966858 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.966712 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-run-systemd\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.966858 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.966770 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-registration-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:51:41.966858 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.966802 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq24z\" (UniqueName: \"kubernetes.io/projected/7af86f87-5d83-4a8b-a0cc-88d4641406d2-kube-api-access-vq24z\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:51:41.966858 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.966834 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-log-socket\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.967152 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.966862 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-sys\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.967152 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.966890 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:51:41.967152 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.966945 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-run-multus-certs\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.967152 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.967009 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk78z\" (UniqueName: \"kubernetes.io/projected/fd6cf358-80f3-42f6-9848-6386a937a57a-kube-api-access-pk78z\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:51:41.967152 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.967054 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-run-netns\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.967152 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.967081 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/95709b19-3c85-4ef1-9110-c23a97847732-ovnkube-config\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.967152 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.967143 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-sysconfig\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.967545 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.967178 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-sysconfig\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.967545 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.967196 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lnnm\" (UniqueName: \"kubernetes.io/projected/e9933e90-d394-4e89-8e0f-115298c359fa-kube-api-access-2lnnm\") pod \"iptables-alerter-db4lm\" (UID: \"e9933e90-d394-4e89-8e0f-115298c359fa\") " pod="openshift-network-operator/iptables-alerter-db4lm" Oct 30 17:51:41.967545 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.967246 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-run-netns\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.967545 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.967285 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-run-systemd\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.967545 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.967381 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-etc-openvswitch\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.967545 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.967406 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-registration-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:51:41.967545 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.967395 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-log-socket\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.967545 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.967452 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e6590e8-bfc4-4b19-b579-7180aa6af22a-kubelet-config\") pod \"global-pull-secret-syncer-5drgz\" (UID: \"1e6590e8-bfc4-4b19-b579-7180aa6af22a\") " pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:51:41.967545 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.967537 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-cnibin\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.967955 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.967580 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-sys\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.967955 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.967663 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:51:41.967955 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.967686 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-cni-netd\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.967955 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.967728 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-cni-netd\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.967955 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.967738 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/95709b19-3c85-4ef1-9110-c23a97847732-ovn-node-metrics-cert\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.967955 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.967793 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f-konnectivity-ca\") pod \"konnectivity-agent-pgghg\" (UID: \"1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f\") " pod="kube-system/konnectivity-agent-pgghg" Oct 30 17:51:41.968235 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.967958 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/95709b19-3c85-4ef1-9110-c23a97847732-ovnkube-config\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.968235 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968004 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/088d4d32-c02b-4da2-9435-c5bc22352079-host\") pod \"node-ca-4hx5j\" (UID: \"088d4d32-c02b-4da2-9435-c5bc22352079\") " pod="openshift-image-registry/node-ca-4hx5j" Oct 30 17:51:41.968235 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968036 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef04039-ef85-40ef-8f30-a0edd7b402e0-tmp-dir\") pod \"node-resolver-pbqhh\" (UID: \"3ef04039-ef85-40ef-8f30-a0edd7b402e0\") " pod="openshift-dns/node-resolver-pbqhh" Oct 30 17:51:41.968235 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968083 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-run-multus-certs\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.968235 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968098 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/088d4d32-c02b-4da2-9435-c5bc22352079-host\") pod \"node-ca-4hx5j\" (UID: \"088d4d32-c02b-4da2-9435-c5bc22352079\") " pod="openshift-image-registry/node-ca-4hx5j" Oct 30 17:51:41.968235 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968135 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-hostroot\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.968235 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fd6cf358-80f3-42f6-9848-6386a937a57a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:51:41.968235 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968213 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e9933e90-d394-4e89-8e0f-115298c359fa-iptables-alerter-script\") pod \"iptables-alerter-db4lm\" (UID: \"e9933e90-d394-4e89-8e0f-115298c359fa\") " pod="openshift-network-operator/iptables-alerter-db4lm" Oct 30 17:51:41.968786 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968238 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-hostroot\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.968786 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968246 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-socket-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:51:41.968786 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968292 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-device-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:51:41.968786 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968363 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fd6cf358-80f3-42f6-9848-6386a937a57a-cni-binary-copy\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:51:41.968786 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968494 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3ef04039-ef85-40ef-8f30-a0edd7b402e0-hosts-file\") pod \"node-resolver-pbqhh\" (UID: \"3ef04039-ef85-40ef-8f30-a0edd7b402e0\") " pod="openshift-dns/node-resolver-pbqhh" Oct 30 17:51:41.968786 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968546 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-cni-binary-copy\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.968786 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968576 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-run-k8s-cni-cncf-io\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.968786 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968609 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-conf-dir\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.968786 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968643 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbs6z\" (UniqueName: \"kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z\") pod \"network-check-target-4xxmr\" (UID: \"90437f91-1657-47db-b3a1-d59bc8d1de14\") " pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:51:41.968786 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968668 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-run\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.968786 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968699 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-var-lib-kubelet\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.968786 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968730 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-var-lib-cni-multus\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.968786 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968731 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef04039-ef85-40ef-8f30-a0edd7b402e0-tmp-dir\") pod \"node-resolver-pbqhh\" (UID: \"3ef04039-ef85-40ef-8f30-a0edd7b402e0\") " pod="openshift-dns/node-resolver-pbqhh" Oct 30 17:51:41.968786 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968761 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-var-lib-kubelet\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.969424 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968877 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-conf-dir\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.969424 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968885 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-kubelet\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.969424 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968937 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-kubelet\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.969424 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968955 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-run-ovn-kubernetes\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.969424 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968987 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-socket-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:51:41.969424 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.968995 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-host\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.969424 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.969054 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-tuned\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.969424 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.969092 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fd6cf358-80f3-42f6-9848-6386a937a57a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:51:41.969424 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.969095 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-sys-fs\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:51:41.969424 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.969127 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f-konnectivity-ca\") pod \"konnectivity-agent-pgghg\" (UID: \"1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f\") " pod="kube-system/konnectivity-agent-pgghg" Oct 30 17:51:41.969424 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.969142 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/088d4d32-c02b-4da2-9435-c5bc22352079-serviceca\") pod \"node-ca-4hx5j\" (UID: \"088d4d32-c02b-4da2-9435-c5bc22352079\") " pod="openshift-image-registry/node-ca-4hx5j" Oct 30 17:51:41.969424 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.969172 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-cni-dir\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.969424 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.969214 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-var-lib-kubelet\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.969424 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.969216 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-var-lib-kubelet\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.969424 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.969286 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-run\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.969424 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.969287 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/95709b19-3c85-4ef1-9110-c23a97847732-env-overrides\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.969424 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.969362 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-cni-dir\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.969424 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.969391 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-var-lib-cni-multus\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.970238 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.969439 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-host\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.970238 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.969439 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-run-ovn-kubernetes\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.970238 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.969551 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-sys-fs\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:51:41.970238 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.969533 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e9933e90-d394-4e89-8e0f-115298c359fa-iptables-alerter-script\") pod \"iptables-alerter-db4lm\" (UID: \"e9933e90-d394-4e89-8e0f-115298c359fa\") " pod="openshift-network-operator/iptables-alerter-db4lm" Oct 30 17:51:41.970238 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.969679 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/95709b19-3c85-4ef1-9110-c23a97847732-env-overrides\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.970238 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.969679 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-device-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:51:41.970238 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.969728 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3ef04039-ef85-40ef-8f30-a0edd7b402e0-hosts-file\") pod \"node-resolver-pbqhh\" (UID: \"3ef04039-ef85-40ef-8f30-a0edd7b402e0\") " pod="openshift-dns/node-resolver-pbqhh" Oct 30 17:51:41.970238 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.969826 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-run-k8s-cni-cncf-io\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.970238 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.969990 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/088d4d32-c02b-4da2-9435-c5bc22352079-serviceca\") pod \"node-ca-4hx5j\" (UID: \"088d4d32-c02b-4da2-9435-c5bc22352079\") " pod="openshift-image-registry/node-ca-4hx5j" Oct 30 17:51:41.970642 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.970548 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-cni-binary-copy\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.971233 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.971209 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-tmp\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.971478 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.971456 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f-agent-certs\") pod \"konnectivity-agent-pgghg\" (UID: \"1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f\") " pod="kube-system/konnectivity-agent-pgghg" Oct 30 17:51:41.971839 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.971818 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-tuned\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.972044 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.972024 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/95709b19-3c85-4ef1-9110-c23a97847732-ovn-node-metrics-cert\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.984485 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.984461 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm6vb\" (UniqueName: \"kubernetes.io/projected/3ef04039-ef85-40ef-8f30-a0edd7b402e0-kube-api-access-lm6vb\") pod \"node-resolver-pbqhh\" (UID: \"3ef04039-ef85-40ef-8f30-a0edd7b402e0\") " pod="openshift-dns/node-resolver-pbqhh" Oct 30 17:51:41.984705 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.984688 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t2rp\" (UniqueName: \"kubernetes.io/projected/088d4d32-c02b-4da2-9435-c5bc22352079-kube-api-access-4t2rp\") pod \"node-ca-4hx5j\" (UID: \"088d4d32-c02b-4da2-9435-c5bc22352079\") " pod="openshift-image-registry/node-ca-4hx5j" Oct 30 17:51:41.984815 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.984799 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ks5l\" (UniqueName: \"kubernetes.io/projected/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-kube-api-access-8ks5l\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:51:41.985568 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.985552 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj9k2\" (UniqueName: \"kubernetes.io/projected/827bbba0-b306-43db-b805-8c87dfabb874-kube-api-access-nj9k2\") pod \"network-metrics-daemon-jnxk7\" (UID: \"827bbba0-b306-43db-b805-8c87dfabb874\") " pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:51:41.986948 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:41.986926 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:51:41.987048 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:41.986953 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:51:41.987048 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:41.986969 2578 projected.go:196] Error preparing data for projected volume kube-api-access-fbs6z for pod openshift-network-diagnostics/network-check-target-4xxmr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:41.987048 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:41.987038 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z podName:90437f91-1657-47db-b3a1-d59bc8d1de14 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:42.487021873 +0000 UTC m=+2.126354679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fbs6z" (UniqueName: "kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z") pod "network-check-target-4xxmr" (UID: "90437f91-1657-47db-b3a1-d59bc8d1de14") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:41.987594 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.987577 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lnnm\" (UniqueName: \"kubernetes.io/projected/e9933e90-d394-4e89-8e0f-115298c359fa-kube-api-access-2lnnm\") pod \"iptables-alerter-db4lm\" (UID: \"e9933e90-d394-4e89-8e0f-115298c359fa\") " pod="openshift-network-operator/iptables-alerter-db4lm" Oct 30 17:51:41.987925 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.987903 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk78z\" (UniqueName: \"kubernetes.io/projected/fd6cf358-80f3-42f6-9848-6386a937a57a-kube-api-access-pk78z\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:51:41.988094 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.988080 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzf62\" (UniqueName: \"kubernetes.io/projected/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-kube-api-access-wzf62\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:41.988489 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.988475 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhlzv\" (UniqueName: \"kubernetes.io/projected/95709b19-3c85-4ef1-9110-c23a97847732-kube-api-access-dhlzv\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:41.993253 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:41.993234 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq24z\" (UniqueName: \"kubernetes.io/projected/7af86f87-5d83-4a8b-a0cc-88d4641406d2-kube-api-access-vq24z\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:51:42.078597 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.078572 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-th8wr" Oct 30 17:51:42.085381 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.085361 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-db4lm" Oct 30 17:51:42.087536 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:42.087494 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471ac5c3_1cdd_4d79_bdc1_3acce27cc042.slice/crio-bfcd9bd03fb23279901627f01b6de1508da2d512c4161a5062b50f5dd429d5ae WatchSource:0}: Error finding container bfcd9bd03fb23279901627f01b6de1508da2d512c4161a5062b50f5dd429d5ae: Status 404 returned error can't find the container with id bfcd9bd03fb23279901627f01b6de1508da2d512c4161a5062b50f5dd429d5ae Oct 30 17:51:42.093479 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.093458 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:51:42.095655 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:42.095632 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9933e90_d394_4e89_8e0f_115298c359fa.slice/crio-c3f671110e035bc547a056600fd7cc028da094c7c56c804695f0eeacab6992e2 WatchSource:0}: Error finding container c3f671110e035bc547a056600fd7cc028da094c7c56c804695f0eeacab6992e2: Status 404 returned error can't find the container with id c3f671110e035bc547a056600fd7cc028da094c7c56c804695f0eeacab6992e2 Oct 30 17:51:42.101382 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.101362 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:51:42.101707 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:42.101689 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af86f87_5d83_4a8b_a0cc_88d4641406d2.slice/crio-d0aa89f80635d0c18e95244519e81178a0752ad297c369bb202fde20a4473346 WatchSource:0}: Error finding container d0aa89f80635d0c18e95244519e81178a0752ad297c369bb202fde20a4473346: Status 404 returned error can't find the container with id d0aa89f80635d0c18e95244519e81178a0752ad297c369bb202fde20a4473346 Oct 30 17:51:42.109222 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.109202 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:51:42.109879 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.109862 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pgghg" Oct 30 17:51:42.110797 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:42.110772 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6cf358_80f3_42f6_9848_6386a937a57a.slice/crio-3f19baba013a313e698e143c78145a5d6e27e42801250497ffdf8688c0de5980 WatchSource:0}: Error finding container 3f19baba013a313e698e143c78145a5d6e27e42801250497ffdf8688c0de5980: Status 404 returned error can't find the container with id 3f19baba013a313e698e143c78145a5d6e27e42801250497ffdf8688c0de5980 Oct 30 17:51:42.113660 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.113553 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:51:42.118157 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:42.118127 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95709b19_3c85_4ef1_9110_c23a97847732.slice/crio-ba0da1b91d88008228eaa5ff170fdec70cf0497ac615c8e1ec2f021f626f49f1 WatchSource:0}: Error finding container ba0da1b91d88008228eaa5ff170fdec70cf0497ac615c8e1ec2f021f626f49f1: Status 404 returned error can't find the container with id ba0da1b91d88008228eaa5ff170fdec70cf0497ac615c8e1ec2f021f626f49f1 Oct 30 17:51:42.119178 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.119153 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4hx5j" Oct 30 17:51:42.121663 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:42.121573 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4bbbf0_cf89_47b9_81cc_9f1e265c2e1f.slice/crio-e3a38c5522ed62d2130d747779834a8925720b6d48468a3d3e0271fc3d656874 WatchSource:0}: Error finding container e3a38c5522ed62d2130d747779834a8925720b6d48468a3d3e0271fc3d656874: Status 404 returned error can't find the container with id e3a38c5522ed62d2130d747779834a8925720b6d48468a3d3e0271fc3d656874 Oct 30 17:51:42.124458 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.124337 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pbqhh" Oct 30 17:51:42.129548 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:42.129526 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod088d4d32_c02b_4da2_9435_c5bc22352079.slice/crio-1b62de9d5d2c99cd9c58cb7336c9808356ca0c869635fbc2db41483bdeeea170 WatchSource:0}: Error finding container 1b62de9d5d2c99cd9c58cb7336c9808356ca0c869635fbc2db41483bdeeea170: Status 404 returned error can't find the container with id 1b62de9d5d2c99cd9c58cb7336c9808356ca0c869635fbc2db41483bdeeea170 Oct 30 17:51:42.134319 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:51:42.134297 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef04039_ef85_40ef_8f30_a0edd7b402e0.slice/crio-da9b5034a6309885bd5ae6b90fbe1356749fb6c830805e9c3387fafacf3aa71a WatchSource:0}: Error finding container da9b5034a6309885bd5ae6b90fbe1356749fb6c830805e9c3387fafacf3aa71a: Status 404 returned error can't find the container with id da9b5034a6309885bd5ae6b90fbe1356749fb6c830805e9c3387fafacf3aa71a Oct 30 17:51:42.473686 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.473654 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret\") pod \"global-pull-secret-syncer-5drgz\" (UID: \"1e6590e8-bfc4-4b19-b579-7180aa6af22a\") " pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:51:42.473864 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.473700 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs\") pod \"network-metrics-daemon-jnxk7\" (UID: \"827bbba0-b306-43db-b805-8c87dfabb874\") " pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:51:42.474019 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:42.474000 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:42.474098 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:42.474071 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret podName:1e6590e8-bfc4-4b19-b579-7180aa6af22a nodeName:}" failed. No retries permitted until 2026-10-30 17:51:43.47405018 +0000 UTC m=+3.113382983 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret") pod "global-pull-secret-syncer-5drgz" (UID: "1e6590e8-bfc4-4b19-b579-7180aa6af22a") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:42.474666 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:42.474645 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:42.474784 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:42.474769 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs podName:827bbba0-b306-43db-b805-8c87dfabb874 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:43.474726156 +0000 UTC m=+3.114058942 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs") pod "network-metrics-daemon-jnxk7" (UID: "827bbba0-b306-43db-b805-8c87dfabb874") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:42.574401 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.574368 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbs6z\" (UniqueName: \"kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z\") pod \"network-check-target-4xxmr\" (UID: \"90437f91-1657-47db-b3a1-d59bc8d1de14\") " pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:51:42.574591 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:42.574558 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:51:42.574591 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:42.574577 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:51:42.574591 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:42.574591 2578 projected.go:196] Error preparing data for projected volume kube-api-access-fbs6z for pod openshift-network-diagnostics/network-check-target-4xxmr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:42.574755 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:42.574643 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z podName:90437f91-1657-47db-b3a1-d59bc8d1de14 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:43.574625537 +0000 UTC m=+3.213958340 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fbs6z" (UniqueName: "kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z") pod "network-check-target-4xxmr" (UID: "90437f91-1657-47db-b3a1-d59bc8d1de14") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:42.579070 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.579012 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:51:42.810890 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.810802 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:46:42 +0000 UTC" deadline="2028-05-28 04:48:29.804769166 +0000 UTC" Oct 30 17:51:42.810890 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.810847 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13810h56m46.993927797s" Oct 30 17:51:42.846512 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.846474 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pbqhh" event={"ID":"3ef04039-ef85-40ef-8f30-a0edd7b402e0","Type":"ContainerStarted","Data":"da9b5034a6309885bd5ae6b90fbe1356749fb6c830805e9c3387fafacf3aa71a"} Oct 30 17:51:42.853840 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.853795 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4hx5j" event={"ID":"088d4d32-c02b-4da2-9435-c5bc22352079","Type":"ContainerStarted","Data":"1b62de9d5d2c99cd9c58cb7336c9808356ca0c869635fbc2db41483bdeeea170"} Oct 30 17:51:42.857243 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.857213 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v5l7g" event={"ID":"fd6cf358-80f3-42f6-9848-6386a937a57a","Type":"ContainerStarted","Data":"3f19baba013a313e698e143c78145a5d6e27e42801250497ffdf8688c0de5980"} Oct 30 17:51:42.868025 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.867994 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-db4lm" event={"ID":"e9933e90-d394-4e89-8e0f-115298c359fa","Type":"ContainerStarted","Data":"c3f671110e035bc547a056600fd7cc028da094c7c56c804695f0eeacab6992e2"} Oct 30 17:51:42.875508 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.875469 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-th8wr" event={"ID":"471ac5c3-1cdd-4d79-bdc1-3acce27cc042","Type":"ContainerStarted","Data":"bfcd9bd03fb23279901627f01b6de1508da2d512c4161a5062b50f5dd429d5ae"} Oct 30 17:51:42.881110 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.881034 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-khz68" event={"ID":"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0","Type":"ContainerStarted","Data":"52fe0aa761a73d30cccc12981410e9b5ee1fb514b7c75262dadae79a3f79e4fe"} Oct 30 17:51:42.890284 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.890257 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pgghg" event={"ID":"1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f","Type":"ContainerStarted","Data":"e3a38c5522ed62d2130d747779834a8925720b6d48468a3d3e0271fc3d656874"} Oct 30 17:51:42.896041 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.895930 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" event={"ID":"95709b19-3c85-4ef1-9110-c23a97847732","Type":"ContainerStarted","Data":"ba0da1b91d88008228eaa5ff170fdec70cf0497ac615c8e1ec2f021f626f49f1"} Oct 30 17:51:42.905056 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.904993 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" event={"ID":"7af86f87-5d83-4a8b-a0cc-88d4641406d2","Type":"ContainerStarted","Data":"d0aa89f80635d0c18e95244519e81178a0752ad297c369bb202fde20a4473346"} Oct 30 17:51:42.952013 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:42.951853 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:51:43.483860 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:43.483819 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret\") pod \"global-pull-secret-syncer-5drgz\" (UID: \"1e6590e8-bfc4-4b19-b579-7180aa6af22a\") " pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:51:43.484063 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:43.483878 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs\") pod \"network-metrics-daemon-jnxk7\" (UID: \"827bbba0-b306-43db-b805-8c87dfabb874\") " pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:51:43.484063 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:43.484043 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:43.484184 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:43.484103 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs podName:827bbba0-b306-43db-b805-8c87dfabb874 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:45.484084648 +0000 UTC m=+5.123417438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs") pod "network-metrics-daemon-jnxk7" (UID: "827bbba0-b306-43db-b805-8c87dfabb874") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:43.484593 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:43.484554 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:43.484699 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:43.484613 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret podName:1e6590e8-bfc4-4b19-b579-7180aa6af22a nodeName:}" failed. No retries permitted until 2026-10-30 17:51:45.484597551 +0000 UTC m=+5.123930355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret") pod "global-pull-secret-syncer-5drgz" (UID: "1e6590e8-bfc4-4b19-b579-7180aa6af22a") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:43.586005 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:43.585201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbs6z\" (UniqueName: \"kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z\") pod \"network-check-target-4xxmr\" (UID: \"90437f91-1657-47db-b3a1-d59bc8d1de14\") " pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:51:43.586005 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:43.585403 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:51:43.586005 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:43.585432 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:51:43.586005 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:43.585447 2578 projected.go:196] Error preparing data for projected volume kube-api-access-fbs6z for pod openshift-network-diagnostics/network-check-target-4xxmr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:43.586005 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:43.585513 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z podName:90437f91-1657-47db-b3a1-d59bc8d1de14 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:45.58549209 +0000 UTC m=+5.224824900 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fbs6z" (UniqueName: "kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z") pod "network-check-target-4xxmr" (UID: "90437f91-1657-47db-b3a1-d59bc8d1de14") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:43.811248 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:43.811128 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:46:42 +0000 UTC" deadline="2028-05-11 21:59:47.019613734 +0000 UTC" Oct 30 17:51:43.811248 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:43.811193 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13420h8m3.208425483s" Oct 30 17:51:43.827792 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:43.827726 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:51:43.827995 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:43.827863 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5drgz" podUID="1e6590e8-bfc4-4b19-b579-7180aa6af22a" Oct 30 17:51:43.828341 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:43.828319 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:51:43.828469 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:43.828445 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnxk7" podUID="827bbba0-b306-43db-b805-8c87dfabb874" Oct 30 17:51:43.828554 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:43.828523 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:51:43.828697 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:43.828633 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxmr" podUID="90437f91-1657-47db-b3a1-d59bc8d1de14" Oct 30 17:51:43.912124 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:43.912045 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:51:45.505715 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:45.505402 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret\") pod \"global-pull-secret-syncer-5drgz\" (UID: \"1e6590e8-bfc4-4b19-b579-7180aa6af22a\") " pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:51:45.505715 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:45.505693 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs\") pod \"network-metrics-daemon-jnxk7\" (UID: \"827bbba0-b306-43db-b805-8c87dfabb874\") " pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:51:45.506123 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:45.505840 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:45.506123 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:45.505905 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs podName:827bbba0-b306-43db-b805-8c87dfabb874 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:49.505884493 +0000 UTC m=+9.145217299 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs") pod "network-metrics-daemon-jnxk7" (UID: "827bbba0-b306-43db-b805-8c87dfabb874") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:45.506381 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:45.506327 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:45.506438 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:45.506391 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret podName:1e6590e8-bfc4-4b19-b579-7180aa6af22a nodeName:}" failed. No retries permitted until 2026-10-30 17:51:49.506373553 +0000 UTC m=+9.145706355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret") pod "global-pull-secret-syncer-5drgz" (UID: "1e6590e8-bfc4-4b19-b579-7180aa6af22a") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:45.607411 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:45.606719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbs6z\" (UniqueName: \"kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z\") pod \"network-check-target-4xxmr\" (UID: \"90437f91-1657-47db-b3a1-d59bc8d1de14\") " pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:51:45.607411 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:45.606949 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:51:45.607411 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:45.606975 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:51:45.607411 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:45.606988 2578 projected.go:196] Error preparing data for projected volume kube-api-access-fbs6z for pod openshift-network-diagnostics/network-check-target-4xxmr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:45.607411 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:45.607049 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z podName:90437f91-1657-47db-b3a1-d59bc8d1de14 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:49.607028212 +0000 UTC m=+9.246361016 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fbs6z" (UniqueName: "kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z") pod "network-check-target-4xxmr" (UID: "90437f91-1657-47db-b3a1-d59bc8d1de14") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:45.827937 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:45.827488 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:51:45.827937 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:45.827637 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5drgz" podUID="1e6590e8-bfc4-4b19-b579-7180aa6af22a" Oct 30 17:51:45.828079 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:45.828045 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:51:45.828403 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:45.828171 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnxk7" podUID="827bbba0-b306-43db-b805-8c87dfabb874" Oct 30 17:51:45.828403 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:45.828227 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:51:45.828403 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:45.828284 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxmr" podUID="90437f91-1657-47db-b3a1-d59bc8d1de14" Oct 30 17:51:45.930264 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:45.930233 2578 generic.go:356] "Generic (PLEG): container finished" podID="bb76617dbf2b01fcc9f68e4666f49e08" containerID="692e7ade818756778142f5833e8debdc2d8988dcfa02d3b2f8e27a58b05c0c56" exitCode=0 Oct 30 17:51:45.930864 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:45.930339 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" event={"ID":"bb76617dbf2b01fcc9f68e4666f49e08","Type":"ContainerDied","Data":"692e7ade818756778142f5833e8debdc2d8988dcfa02d3b2f8e27a58b05c0c56"} Oct 30 17:51:45.942718 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:45.942125 2578 generic.go:356] "Generic (PLEG): container finished" podID="fd6cf358-80f3-42f6-9848-6386a937a57a" containerID="5ff56b6c3362d55bb9f72d5615a9ab7dafa5de1e2010c0a56e0c97bc6b34016b" exitCode=0 Oct 30 17:51:45.942718 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:45.942161 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v5l7g" event={"ID":"fd6cf358-80f3-42f6-9848-6386a937a57a","Type":"ContainerDied","Data":"5ff56b6c3362d55bb9f72d5615a9ab7dafa5de1e2010c0a56e0c97bc6b34016b"} Oct 30 17:51:46.952527 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:46.952481 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" event={"ID":"bb76617dbf2b01fcc9f68e4666f49e08","Type":"ContainerStarted","Data":"4a295c8c490d1136c64f86524dc2109bd153ae24df1ea79b2edd4c039daae0a1"} Oct 30 17:51:46.956864 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:46.956833 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal" event={"ID":"f50f91e38dadbc11b70b476e358bc37f","Type":"ContainerStarted","Data":"c472835fdfec63debb2385e7e792c37045717452ab7cdf7fe0b8384c78fc6fa5"} Oct 30 17:51:46.964278 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:46.964252 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pgghg" event={"ID":"1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f","Type":"ContainerStarted","Data":"ab8d42e898c9c9a318837163f2842e688328bdd066e5d79d3a14d6227f3b8da2"} Oct 30 17:51:46.991823 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:46.991770 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" podStartSLOduration=5.991754391 podStartE2EDuration="5.991754391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:46.991119358 +0000 UTC m=+6.630452180" watchObservedRunningTime="2026-10-30 17:51:46.991754391 +0000 UTC m=+6.631087204" Oct 30 17:51:47.068251 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:47.068197 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pgghg" podStartSLOduration=1.768643447 podStartE2EDuration="6.0681782s" totalImagesPullingTime="4.299534753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:47.03113785 +0000 UTC m=+6.670470666" watchObservedRunningTime="2026-10-30 17:51:47.0681782 +0000 UTC m=+6.707511012" Oct 30 17:51:47.255431 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:47.255205 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pgghg" Oct 30 17:51:47.255766 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:47.255747 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pgghg" Oct 30 17:51:47.294408 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:47.294074 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal" podStartSLOduration=6.294059885 podStartE2EDuration="6.294059885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:47.06716394 +0000 UTC m=+6.706496788" watchObservedRunningTime="2026-10-30 17:51:47.294059885 +0000 UTC m=+6.933392695" Oct 30 17:51:47.827149 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:47.827117 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:51:47.827370 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:47.827198 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:51:47.827370 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:47.827224 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:51:47.827485 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:47.827373 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5drgz" podUID="1e6590e8-bfc4-4b19-b579-7180aa6af22a" Oct 30 17:51:47.827556 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:47.827533 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxmr" podUID="90437f91-1657-47db-b3a1-d59bc8d1de14" Oct 30 17:51:47.827648 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:47.827625 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnxk7" podUID="827bbba0-b306-43db-b805-8c87dfabb874" Oct 30 17:51:48.974739 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:48.973875 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4hx5j" event={"ID":"088d4d32-c02b-4da2-9435-c5bc22352079","Type":"ContainerStarted","Data":"eb27c74f30064383a90d9fc46a836776f331ff29525b7d0686d6ccb541ac592a"} Oct 30 17:51:48.977580 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:48.977062 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:51:48.977580 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:48.977486 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" event={"ID":"7af86f87-5d83-4a8b-a0cc-88d4641406d2","Type":"ContainerStarted","Data":"15e88ddeaf73f4798de761c59e5d135e1dbca56433fee2cee942028183d5dede"} Oct 30 17:51:49.542123 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:49.541266 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret\") pod \"global-pull-secret-syncer-5drgz\" (UID: \"1e6590e8-bfc4-4b19-b579-7180aa6af22a\") " pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:51:49.542123 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:49.541322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs\") pod \"network-metrics-daemon-jnxk7\" (UID: \"827bbba0-b306-43db-b805-8c87dfabb874\") " pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:51:49.542123 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:49.541509 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:49.542123 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:49.541595 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs podName:827bbba0-b306-43db-b805-8c87dfabb874 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:57.541573364 +0000 UTC m=+17.180906169 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs") pod "network-metrics-daemon-jnxk7" (UID: "827bbba0-b306-43db-b805-8c87dfabb874") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:49.542123 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:49.542034 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:49.542123 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:49.542086 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret podName:1e6590e8-bfc4-4b19-b579-7180aa6af22a nodeName:}" failed. No retries permitted until 2026-10-30 17:51:57.542072328 +0000 UTC m=+17.181405131 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret") pod "global-pull-secret-syncer-5drgz" (UID: "1e6590e8-bfc4-4b19-b579-7180aa6af22a") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:49.642767 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:49.642653 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbs6z\" (UniqueName: \"kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z\") pod \"network-check-target-4xxmr\" (UID: \"90437f91-1657-47db-b3a1-d59bc8d1de14\") " pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:51:49.642953 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:49.642883 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:51:49.642953 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:49.642903 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:51:49.642953 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:49.642917 2578 projected.go:196] Error preparing data for projected volume kube-api-access-fbs6z for pod openshift-network-diagnostics/network-check-target-4xxmr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:49.643121 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:49.642975 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z podName:90437f91-1657-47db-b3a1-d59bc8d1de14 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:57.642957127 +0000 UTC m=+17.282289921 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fbs6z" (UniqueName: "kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z") pod "network-check-target-4xxmr" (UID: "90437f91-1657-47db-b3a1-d59bc8d1de14") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:49.827901 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:49.827866 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:51:49.828084 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:49.828001 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5drgz" podUID="1e6590e8-bfc4-4b19-b579-7180aa6af22a" Oct 30 17:51:49.828360 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:49.828342 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:51:49.828468 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:49.828424 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnxk7" podUID="827bbba0-b306-43db-b805-8c87dfabb874" Oct 30 17:51:49.828468 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:49.828464 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:51:49.828591 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:49.828507 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxmr" podUID="90437f91-1657-47db-b3a1-d59bc8d1de14" Oct 30 17:51:50.385564 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:50.385530 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 17:51:50.834084 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:50.833961 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T17:51:50.385556489Z","UUID":"ff45c530-c74b-419b-ac79-dd99687e0893","Handler":null,"Name":"","Endpoint":""} Oct 30 17:51:50.836843 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:50.836531 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 17:51:50.836843 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:50.836559 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 17:51:50.985927 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:50.985615 2578 generic.go:356] "Generic (PLEG): container finished" podID="fd6cf358-80f3-42f6-9848-6386a937a57a" containerID="1aa632c64fb0b1a8bd30a1fcba31675fa108240401d3604dc4ba351de7431ac5" exitCode=0 Oct 30 17:51:50.985927 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:50.985854 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v5l7g" event={"ID":"fd6cf358-80f3-42f6-9848-6386a937a57a","Type":"ContainerDied","Data":"1aa632c64fb0b1a8bd30a1fcba31675fa108240401d3604dc4ba351de7431ac5"} Oct 30 17:51:50.988006 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:50.987978 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-th8wr" event={"ID":"471ac5c3-1cdd-4d79-bdc1-3acce27cc042","Type":"ContainerStarted","Data":"484158e6d152b06406cd1b1eda44eb68457198b177f5a8b5ff211ae7e61701b1"} Oct 30 17:51:50.992694 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:50.992665 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" event={"ID":"7af86f87-5d83-4a8b-a0cc-88d4641406d2","Type":"ContainerStarted","Data":"bd1c76bc16664518a6d8266bcb1a68c28ee20d00f6580257d22146a24a4fc31f"} Oct 30 17:51:51.046217 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:51.046156 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4hx5j" podStartSLOduration=4.117768497 podStartE2EDuration="10.046133775s" totalImagesPullingTime="5.928365278s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:49.013220904 +0000 UTC m=+8.652553718" watchObservedRunningTime="2026-10-30 17:51:51.046133775 +0000 UTC m=+10.685466589" Oct 30 17:51:51.092344 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:51.092292 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-th8wr" podStartSLOduration=1.6050059079999999 podStartE2EDuration="10.09227414s" totalImagesPullingTime="8.487268232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:51.091053788 +0000 UTC m=+10.730386600" watchObservedRunningTime="2026-10-30 17:51:51.09227414 +0000 UTC m=+10.731606946" Oct 30 17:51:51.828022 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:51.827480 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:51:51.828022 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:51.827491 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:51:51.828022 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:51.827641 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:51:51.828022 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:51.827639 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxmr" podUID="90437f91-1657-47db-b3a1-d59bc8d1de14" Oct 30 17:51:51.828022 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:51.827758 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnxk7" podUID="827bbba0-b306-43db-b805-8c87dfabb874" Oct 30 17:51:51.828022 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:51.827839 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5drgz" podUID="1e6590e8-bfc4-4b19-b579-7180aa6af22a" Oct 30 17:51:51.996819 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:51.996724 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pbqhh" event={"ID":"3ef04039-ef85-40ef-8f30-a0edd7b402e0","Type":"ContainerStarted","Data":"c7d7daf8bf4e38885f0d5ed501db86019e005e704746f7a7aa47ce00eb1eb120"} Oct 30 17:51:51.999469 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:51.999434 2578 generic.go:356] "Generic (PLEG): container finished" podID="fd6cf358-80f3-42f6-9848-6386a937a57a" containerID="2e8440a41a16e5aa29265e6f7118ed5944a7c021e0dd32a24184ba92c5c4050e" exitCode=0 Oct 30 17:51:51.999646 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:51.999523 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v5l7g" event={"ID":"fd6cf358-80f3-42f6-9848-6386a937a57a","Type":"ContainerDied","Data":"2e8440a41a16e5aa29265e6f7118ed5944a7c021e0dd32a24184ba92c5c4050e"} Oct 30 17:51:52.002700 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:52.002638 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" event={"ID":"7af86f87-5d83-4a8b-a0cc-88d4641406d2","Type":"ContainerStarted","Data":"8b5c8e3f337be83d94a750fac43f041aa67e77857f294b0d0b09c042f96604ca"} Oct 30 17:51:52.034513 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:52.034454 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-pbqhh" podStartSLOduration=2.086416881 podStartE2EDuration="11.034439605s" totalImagesPullingTime="8.948022724s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:52.033906758 +0000 UTC m=+11.673239570" watchObservedRunningTime="2026-10-30 17:51:52.034439605 +0000 UTC m=+11.673772416" Oct 30 17:51:52.074917 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:52.074048 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" podStartSLOduration=2.23639761 podStartE2EDuration="11.074031518s" totalImagesPullingTime="8.837633908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:52.073702933 +0000 UTC m=+11.713035746" watchObservedRunningTime="2026-10-30 17:51:52.074031518 +0000 UTC m=+11.713364333" Oct 30 17:51:53.006820 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:53.006739 2578 generic.go:356] "Generic (PLEG): container finished" podID="fd6cf358-80f3-42f6-9848-6386a937a57a" containerID="4ecd8efb85c6e2ca08221d8959859ac10afcbc615e6cc870a87e578348b00756" exitCode=0 Oct 30 17:51:53.007258 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:53.006810 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v5l7g" event={"ID":"fd6cf358-80f3-42f6-9848-6386a937a57a","Type":"ContainerDied","Data":"4ecd8efb85c6e2ca08221d8959859ac10afcbc615e6cc870a87e578348b00756"} Oct 30 17:51:53.008984 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:53.008549 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-db4lm" event={"ID":"e9933e90-d394-4e89-8e0f-115298c359fa","Type":"ContainerStarted","Data":"603959b13b87970a98815c3092a7012bca8b522ddbd9de7a8849e40d5392664b"} Oct 30 17:51:53.091118 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:53.091026 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-db4lm" podStartSLOduration=3.258860418 podStartE2EDuration="12.09100989s" totalImagesPullingTime="8.832149472s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:53.090842178 +0000 UTC m=+12.730174987" watchObservedRunningTime="2026-10-30 17:51:53.09100989 +0000 UTC m=+12.730342699" Oct 30 17:51:53.827728 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:53.827673 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:51:53.827911 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:53.827743 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:51:53.827911 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:53.827847 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:51:53.827911 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:53.827854 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxmr" podUID="90437f91-1657-47db-b3a1-d59bc8d1de14" Oct 30 17:51:53.828062 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:53.827951 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5drgz" podUID="1e6590e8-bfc4-4b19-b579-7180aa6af22a" Oct 30 17:51:53.828062 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:53.828015 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnxk7" podUID="827bbba0-b306-43db-b805-8c87dfabb874" Oct 30 17:51:54.012883 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:54.012836 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-khz68" event={"ID":"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0","Type":"ContainerStarted","Data":"66a750d5a12aa0165b3c41ecf46c5db8d241c482b357f4cdc56288d90b7354f3"} Oct 30 17:51:54.050483 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:54.050435 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-khz68" podStartSLOduration=2.003115607 podStartE2EDuration="13.050416839s" totalImagesPullingTime="11.047301232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:54.050300146 +0000 UTC m=+13.689632955" watchObservedRunningTime="2026-10-30 17:51:54.050416839 +0000 UTC m=+13.689749649" Oct 30 17:51:55.019193 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:55.018866 2578 generic.go:356] "Generic (PLEG): container finished" podID="fd6cf358-80f3-42f6-9848-6386a937a57a" containerID="1e81aa2a56deaea12669bbd85f3ff99a60dd79569a82838af62f9c7032265de3" exitCode=0 Oct 30 17:51:55.019193 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:55.018941 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v5l7g" event={"ID":"fd6cf358-80f3-42f6-9848-6386a937a57a","Type":"ContainerDied","Data":"1e81aa2a56deaea12669bbd85f3ff99a60dd79569a82838af62f9c7032265de3"} Oct 30 17:51:55.827459 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:55.827427 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:51:55.827680 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:55.827428 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:51:55.827680 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:55.827440 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:51:55.827680 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:55.827585 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnxk7" podUID="827bbba0-b306-43db-b805-8c87dfabb874" Oct 30 17:51:55.827829 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:55.827706 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxmr" podUID="90437f91-1657-47db-b3a1-d59bc8d1de14" Oct 30 17:51:55.827829 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:55.827788 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5drgz" podUID="1e6590e8-bfc4-4b19-b579-7180aa6af22a" Oct 30 17:51:56.024872 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:56.024837 2578 generic.go:356] "Generic (PLEG): container finished" podID="fd6cf358-80f3-42f6-9848-6386a937a57a" containerID="42762e72f271737907c69aa129ffa700e811db4ae9a47b10e2da93fd28aac9c4" exitCode=0 Oct 30 17:51:56.025275 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:56.024889 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v5l7g" event={"ID":"fd6cf358-80f3-42f6-9848-6386a937a57a","Type":"ContainerDied","Data":"42762e72f271737907c69aa129ffa700e811db4ae9a47b10e2da93fd28aac9c4"} Oct 30 17:51:57.030643 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:57.030607 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v5l7g" event={"ID":"fd6cf358-80f3-42f6-9848-6386a937a57a","Type":"ContainerStarted","Data":"a59dc7ddbf3da99679072b9efac352da0c775774adcdb64c9504defce6f4dd4e"} Oct 30 17:51:57.108246 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:57.108197 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v5l7g" podStartSLOduration=5.416972682 podStartE2EDuration="16.108184793s" totalImagesPullingTime="10.691212111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:57.107519801 +0000 UTC m=+16.746852602" watchObservedRunningTime="2026-10-30 17:51:57.108184793 +0000 UTC m=+16.747517601" Oct 30 17:51:57.612089 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:57.612059 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret\") pod \"global-pull-secret-syncer-5drgz\" (UID: \"1e6590e8-bfc4-4b19-b579-7180aa6af22a\") " pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:51:57.612089 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:57.612092 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs\") pod \"network-metrics-daemon-jnxk7\" (UID: \"827bbba0-b306-43db-b805-8c87dfabb874\") " pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:51:57.612282 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:57.612185 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:57.612282 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:57.612186 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:57.612282 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:57.612230 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs podName:827bbba0-b306-43db-b805-8c87dfabb874 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:13.612218834 +0000 UTC m=+33.251551634 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs") pod "network-metrics-daemon-jnxk7" (UID: "827bbba0-b306-43db-b805-8c87dfabb874") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:57.612282 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:57.612243 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret podName:1e6590e8-bfc4-4b19-b579-7180aa6af22a nodeName:}" failed. No retries permitted until 2026-10-30 17:52:13.612237134 +0000 UTC m=+33.251569920 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret") pod "global-pull-secret-syncer-5drgz" (UID: "1e6590e8-bfc4-4b19-b579-7180aa6af22a") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:57.713245 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:57.713222 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbs6z\" (UniqueName: \"kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z\") pod \"network-check-target-4xxmr\" (UID: \"90437f91-1657-47db-b3a1-d59bc8d1de14\") " pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:51:57.713375 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:57.713360 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:51:57.713420 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:57.713382 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:51:57.713420 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:57.713391 2578 projected.go:196] Error preparing data for projected volume kube-api-access-fbs6z for pod openshift-network-diagnostics/network-check-target-4xxmr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:57.713482 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:57.713429 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z podName:90437f91-1657-47db-b3a1-d59bc8d1de14 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:13.713417687 +0000 UTC m=+33.352750486 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fbs6z" (UniqueName: "kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z") pod "network-check-target-4xxmr" (UID: "90437f91-1657-47db-b3a1-d59bc8d1de14") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:57.827652 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:57.827630 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:51:57.827735 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:57.827647 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:51:57.827735 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:57.827630 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:51:57.827822 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:57.827731 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5drgz" podUID="1e6590e8-bfc4-4b19-b579-7180aa6af22a" Oct 30 17:51:57.827822 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:57.827778 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxmr" podUID="90437f91-1657-47db-b3a1-d59bc8d1de14" Oct 30 17:51:57.827885 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:57.827835 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnxk7" podUID="827bbba0-b306-43db-b805-8c87dfabb874" Oct 30 17:51:58.037422 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:58.037286 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 17:51:58.037911 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:58.037867 2578 generic.go:356] "Generic (PLEG): container finished" podID="95709b19-3c85-4ef1-9110-c23a97847732" containerID="12088b44ca44e4d059d866fe4f913661c812ad6da59b28ceecc54487e0efdf6c" exitCode=1 Oct 30 17:51:58.037952 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:58.037924 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" event={"ID":"95709b19-3c85-4ef1-9110-c23a97847732","Type":"ContainerStarted","Data":"7fa96a37e0049670801fb2610fb9492a0d7bf09085d164487180adc336a2f171"} Oct 30 17:51:58.038047 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:58.038018 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" event={"ID":"95709b19-3c85-4ef1-9110-c23a97847732","Type":"ContainerStarted","Data":"417aa7332b5230a12f04984ecc69decc744a83f662fda5cdc23278bfa5fc24ef"} Oct 30 17:51:58.038090 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:58.038079 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" event={"ID":"95709b19-3c85-4ef1-9110-c23a97847732","Type":"ContainerStarted","Data":"9c7c27c86d9cebf5c5aa52fd3b7b390ee08644939b7ee260b490f2747e144666"} Oct 30 17:51:58.038121 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:58.038109 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" event={"ID":"95709b19-3c85-4ef1-9110-c23a97847732","Type":"ContainerDied","Data":"12088b44ca44e4d059d866fe4f913661c812ad6da59b28ceecc54487e0efdf6c"} Oct 30 17:51:58.038150 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:58.038139 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" event={"ID":"95709b19-3c85-4ef1-9110-c23a97847732","Type":"ContainerStarted","Data":"40119f820782ad585f06702996d91d92b2d60a0060cb0959014a0e12360e1028"} Oct 30 17:51:59.043113 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:59.043086 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 17:51:59.043631 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:59.043485 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" event={"ID":"95709b19-3c85-4ef1-9110-c23a97847732","Type":"ContainerStarted","Data":"49b7f2aa6cf6cef30cb74defb3888e30d1b1d3053d7b061ad1d7f5b8d0ae933d"} Oct 30 17:51:59.827658 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:59.827626 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:51:59.827658 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:59.827658 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:51:59.827859 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:51:59.827751 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:51:59.827859 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:59.827753 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnxk7" podUID="827bbba0-b306-43db-b805-8c87dfabb874" Oct 30 17:51:59.827859 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:59.827827 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5drgz" podUID="1e6590e8-bfc4-4b19-b579-7180aa6af22a" Oct 30 17:51:59.827951 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:51:59.827869 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxmr" podUID="90437f91-1657-47db-b3a1-d59bc8d1de14" Oct 30 17:52:01.049426 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:01.049397 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 17:52:01.049907 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:01.049720 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" event={"ID":"95709b19-3c85-4ef1-9110-c23a97847732","Type":"ContainerStarted","Data":"5b57caa3b82c79d3b6517ee07d8db0ae5ad05af9dc823964d2a2c859530a0b2d"} Oct 30 17:52:01.827585 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:01.827554 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:52:01.827585 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:01.827569 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:52:01.827784 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:01.827678 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5drgz" podUID="1e6590e8-bfc4-4b19-b579-7180aa6af22a" Oct 30 17:52:01.827784 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:01.827726 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:52:01.827847 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:01.827809 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnxk7" podUID="827bbba0-b306-43db-b805-8c87dfabb874" Oct 30 17:52:01.827895 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:01.827869 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxmr" podUID="90437f91-1657-47db-b3a1-d59bc8d1de14" Oct 30 17:52:03.056227 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:03.056202 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 17:52:03.056848 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:03.056583 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" event={"ID":"95709b19-3c85-4ef1-9110-c23a97847732","Type":"ContainerStarted","Data":"eff7b18a184007f081ca0196a70600ba5005275256be095d98c65cfd44ddcfb4"} Oct 30 17:52:03.056897 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:03.056847 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:03.056948 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:03.056928 2578 scope.go:122] "RemoveContainer" containerID="12088b44ca44e4d059d866fe4f913661c812ad6da59b28ceecc54487e0efdf6c" Oct 30 17:52:03.074117 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:03.073905 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:03.828025 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:03.827994 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:52:03.828271 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:03.828029 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:52:03.828271 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:03.828089 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnxk7" podUID="827bbba0-b306-43db-b805-8c87dfabb874" Oct 30 17:52:03.828271 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:03.828126 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:52:03.828271 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:03.828196 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxmr" podUID="90437f91-1657-47db-b3a1-d59bc8d1de14" Oct 30 17:52:03.828424 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:03.828277 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5drgz" podUID="1e6590e8-bfc4-4b19-b579-7180aa6af22a" Oct 30 17:52:04.062486 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:04.062463 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 17:52:04.062906 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:04.062857 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" event={"ID":"95709b19-3c85-4ef1-9110-c23a97847732","Type":"ContainerStarted","Data":"fa41f4a7f115f4a9594fda7dbd5d05a77de9e9d36ba1df14419811de8d3983cc"} Oct 30 17:52:04.063088 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:04.063066 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:04.063229 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:04.063149 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:04.078910 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:04.078858 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:04.141338 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:04.141295 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" podStartSLOduration=7.5654405350000005 podStartE2EDuration="23.141282366s" totalImagesPullingTime="15.575841831s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:04.140874361 +0000 UTC m=+23.780207170" watchObservedRunningTime="2026-10-30 17:52:04.141282366 +0000 UTC m=+23.780615190" Oct 30 17:52:04.295371 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:04.295345 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5drgz"] Oct 30 17:52:04.295538 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:04.295460 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:52:04.295605 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:04.295588 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5drgz" podUID="1e6590e8-bfc4-4b19-b579-7180aa6af22a" Oct 30 17:52:04.301078 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:04.301053 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4xxmr"] Oct 30 17:52:04.301195 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:04.301141 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:52:04.301263 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:04.301239 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxmr" podUID="90437f91-1657-47db-b3a1-d59bc8d1de14" Oct 30 17:52:04.301642 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:04.301619 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jnxk7"] Oct 30 17:52:04.301752 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:04.301715 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:52:04.301844 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:04.301817 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnxk7" podUID="827bbba0-b306-43db-b805-8c87dfabb874" Oct 30 17:52:05.827810 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:05.827778 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:52:05.828247 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:05.827778 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:52:05.828247 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:05.827979 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxmr" podUID="90437f91-1657-47db-b3a1-d59bc8d1de14" Oct 30 17:52:05.828247 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:05.827873 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5drgz" podUID="1e6590e8-bfc4-4b19-b579-7180aa6af22a" Oct 30 17:52:05.828247 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:05.827779 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:52:05.828247 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:05.828101 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnxk7" podUID="827bbba0-b306-43db-b805-8c87dfabb874" Oct 30 17:52:07.827711 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:07.827676 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:52:07.828112 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:07.827679 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:52:07.828112 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:07.827780 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnxk7" podUID="827bbba0-b306-43db-b805-8c87dfabb874" Oct 30 17:52:07.828112 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:07.827676 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:52:07.828112 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:07.827911 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxmr" podUID="90437f91-1657-47db-b3a1-d59bc8d1de14" Oct 30 17:52:07.828112 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:07.827845 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5drgz" podUID="1e6590e8-bfc4-4b19-b579-7180aa6af22a" Oct 30 17:52:09.827002 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:09.826975 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:52:09.827002 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:09.826989 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:52:09.827366 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:09.827067 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxmr" podUID="90437f91-1657-47db-b3a1-d59bc8d1de14" Oct 30 17:52:09.827366 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:09.827097 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:52:09.827366 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:09.827179 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnxk7" podUID="827bbba0-b306-43db-b805-8c87dfabb874" Oct 30 17:52:09.827366 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:09.827255 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5drgz" podUID="1e6590e8-bfc4-4b19-b579-7180aa6af22a" Oct 30 17:52:10.125207 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.125021 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeReady" Oct 30 17:52:10.125300 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.125244 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 17:52:10.200130 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.200098 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-kkgjk"] Oct 30 17:52:10.221995 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.221969 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64bf8b8bb7-dlchr"] Oct 30 17:52:10.222116 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.222097 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-kkgjk" Oct 30 17:52:10.231655 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.231632 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-w725f"] Oct 30 17:52:10.231762 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.231736 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.237794 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.237774 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf"] Oct 30 17:52:10.237910 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.237895 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-w725f" Oct 30 17:52:10.250041 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.250021 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69885b9d-n2bmz"] Oct 30 17:52:10.250136 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.250121 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf" Oct 30 17:52:10.252184 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.252168 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6xn6r"] Oct 30 17:52:10.252260 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.252236 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.258181 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.258156 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-kkgjk"] Oct 30 17:52:10.258266 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.258250 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6xn6r" Oct 30 17:52:10.263641 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.263616 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-w725f"] Oct 30 17:52:10.271636 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.271613 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf"] Oct 30 17:52:10.296030 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.296006 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7r5h8"] Oct 30 17:52:10.298734 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.298716 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:10.299264 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.299246 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7txkr"] Oct 30 17:52:10.300319 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.300300 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkgjk\" (UID: \"f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkgjk" Oct 30 17:52:10.300414 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.300330 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kkgjk\" (UID: \"f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkgjk" Oct 30 17:52:10.302746 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.302731 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7txkr" Oct 30 17:52:10.303565 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.303534 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64bf8b8bb7-dlchr"] Oct 30 17:52:10.310445 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.310421 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69885b9d-n2bmz"] Oct 30 17:52:10.321394 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.321377 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7r5h8"] Oct 30 17:52:10.326417 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.326395 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7txkr"] Oct 30 17:52:10.344952 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.344934 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 17:52:10.345032 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.344979 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 17:52:10.345032 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.344995 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 17:52:10.346338 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.346321 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 17:52:10.346497 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.346482 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 17:52:10.346799 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.346784 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-wfmdw\"" Oct 30 17:52:10.346879 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.346864 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 17:52:10.346930 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.346873 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6xmc6\"" Oct 30 17:52:10.346975 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.346935 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vwtrh\"" Oct 30 17:52:10.347487 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.347473 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 17:52:10.347597 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.347585 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 17:52:10.355188 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.355169 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 17:52:10.358045 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.358024 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 17:52:10.358145 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.358089 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 17:52:10.358145 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.358111 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 17:52:10.364316 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.364289 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 17:52:10.364316 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.364290 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:52:10.364316 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.364311 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-96ztk\"" Oct 30 17:52:10.364576 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.364317 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 17:52:10.364576 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.364359 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-99rd9\"" Oct 30 17:52:10.365175 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.365156 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6xn6r"] Oct 30 17:52:10.365271 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.365257 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 17:52:10.365330 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.365262 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 17:52:10.365400 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.365387 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-zd7pm\"" Oct 30 17:52:10.365458 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.365401 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 17:52:10.365458 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.365444 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 17:52:10.401202 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.401151 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-ca-trust-extracted\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.401202 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.401175 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21a54420-0e75-4d4b-93cd-fe72c0cc2070-installation-pull-secrets\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.401202 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.401196 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-image-registry-private-configuration\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.401349 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.401226 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8e818352-df23-46b4-84d9-81fd259200ec-config-volume\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:10.401349 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.401287 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-certificates\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.401349 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.401327 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21a54420-0e75-4d4b-93cd-fe72c0cc2070-image-registry-private-configuration\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.401442 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.401430 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgxhb\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-kube-api-access-vgxhb\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.401471 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.401446 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9602b28-165a-4c17-96fa-9f0e25fd2cf8-cert\") pod \"ingress-canary-7txkr\" (UID: \"c9602b28-165a-4c17-96fa-9f0e25fd2cf8\") " pod="openshift-ingress-canary/ingress-canary-7txkr" Oct 30 17:52:10.401471 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.401464 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:10.401564 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.401482 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-trusted-ca\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.401564 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.401522 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-bound-sa-token\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.401564 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.401546 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmwhl\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-kube-api-access-hmwhl\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.401671 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.401600 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4bqp\" (UniqueName: \"kubernetes.io/projected/355e77f7-f944-4955-95c2-bdf4c7ea12b7-kube-api-access-d4bqp\") pod \"kas-connection-checker-685895c7c9-6xn6r\" (UID: \"355e77f7-f944-4955-95c2-bdf4c7ea12b7\") " pod="kube-system/kas-connection-checker-685895c7c9-6xn6r" Oct 30 17:52:10.401671 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.401626 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21a54420-0e75-4d4b-93cd-fe72c0cc2070-ca-trust-extracted\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.401671 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.401659 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf\" (UID: \"b4d17d89-b883-472d-84c9-b8f4d0c8eb7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf" Oct 30 17:52:10.401769 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.401678 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-tls\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.401769 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.401697 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-tls\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.401769 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.401719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkgjk\" (UID: \"f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkgjk" Oct 30 17:52:10.401769 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.401739 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbc66\" (UniqueName: \"kubernetes.io/projected/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e-kube-api-access-qbc66\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf\" (UID: \"b4d17d89-b883-472d-84c9-b8f4d0c8eb7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf" Oct 30 17:52:10.401896 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.401765 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll92d\" (UniqueName: \"kubernetes.io/projected/d58a06a1-fdbc-439e-8572-fdeabc6ee4ef-kube-api-access-ll92d\") pod \"network-check-source-6f78d59f99-w725f\" (UID: \"d58a06a1-fdbc-439e-8572-fdeabc6ee4ef\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-w725f" Oct 30 17:52:10.401896 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:10.401834 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:52:10.401896 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.401839 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9xxt\" (UniqueName: \"kubernetes.io/projected/c9602b28-165a-4c17-96fa-9f0e25fd2cf8-kube-api-access-r9xxt\") pod \"ingress-canary-7txkr\" (UID: \"c9602b28-165a-4c17-96fa-9f0e25fd2cf8\") " pod="openshift-ingress-canary/ingress-canary-7txkr" Oct 30 17:52:10.401896 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:10.401879 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-networking-console-plugin-cert podName:f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:10.901867219 +0000 UTC m=+30.541200032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kkgjk" (UID: "f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2") : secret "networking-console-plugin-cert" not found Oct 30 17:52:10.402046 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.401913 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kkgjk\" (UID: \"f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkgjk" Oct 30 17:52:10.402046 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.401947 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21a54420-0e75-4d4b-93cd-fe72c0cc2070-trusted-ca\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.402117 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.402066 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf\" (UID: \"b4d17d89-b883-472d-84c9-b8f4d0c8eb7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf" Oct 30 17:52:10.402162 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.402121 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-bound-sa-token\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.402196 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.402176 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e818352-df23-46b4-84d9-81fd259200ec-tmp-dir\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:10.402244 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.402230 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-installation-pull-secrets\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.402448 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.402251 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-certificates\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.402448 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.402269 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qplp\" (UniqueName: \"kubernetes.io/projected/8e818352-df23-46b4-84d9-81fd259200ec-kube-api-access-4qplp\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:10.402780 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.402765 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kkgjk\" (UID: \"f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkgjk" Oct 30 17:52:10.502557 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.502533 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21a54420-0e75-4d4b-93cd-fe72c0cc2070-trusted-ca\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.502673 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.502563 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf\" (UID: \"b4d17d89-b883-472d-84c9-b8f4d0c8eb7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf" Oct 30 17:52:10.502673 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.502582 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-bound-sa-token\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.502673 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.502627 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e818352-df23-46b4-84d9-81fd259200ec-tmp-dir\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:10.502673 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.502657 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-installation-pull-secrets\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.502823 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.502681 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-certificates\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.503331 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.503298 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e818352-df23-46b4-84d9-81fd259200ec-tmp-dir\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:10.503789 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.503759 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qplp\" (UniqueName: \"kubernetes.io/projected/8e818352-df23-46b4-84d9-81fd259200ec-kube-api-access-4qplp\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:10.503887 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.503814 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-ca-trust-extracted\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.503887 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.503847 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21a54420-0e75-4d4b-93cd-fe72c0cc2070-installation-pull-secrets\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.503887 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.503875 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-image-registry-private-configuration\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.504033 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.503907 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8e818352-df23-46b4-84d9-81fd259200ec-config-volume\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:10.504033 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.503949 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-certificates\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.504033 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.503980 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21a54420-0e75-4d4b-93cd-fe72c0cc2070-image-registry-private-configuration\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.504170 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.504076 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgxhb\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-kube-api-access-vgxhb\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.504170 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.504106 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9602b28-165a-4c17-96fa-9f0e25fd2cf8-cert\") pod \"ingress-canary-7txkr\" (UID: \"c9602b28-165a-4c17-96fa-9f0e25fd2cf8\") " pod="openshift-ingress-canary/ingress-canary-7txkr" Oct 30 17:52:10.504170 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.504160 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:10.504336 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.503810 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-certificates\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.504394 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.504322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-trusted-ca\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.504597 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.504395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-bound-sa-token\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.504668 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.504624 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmwhl\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-kube-api-access-hmwhl\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.504721 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.504669 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4bqp\" (UniqueName: \"kubernetes.io/projected/355e77f7-f944-4955-95c2-bdf4c7ea12b7-kube-api-access-d4bqp\") pod \"kas-connection-checker-685895c7c9-6xn6r\" (UID: \"355e77f7-f944-4955-95c2-bdf4c7ea12b7\") " pod="kube-system/kas-connection-checker-685895c7c9-6xn6r" Oct 30 17:52:10.504773 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.504721 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21a54420-0e75-4d4b-93cd-fe72c0cc2070-ca-trust-extracted\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.504918 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.504756 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf\" (UID: \"b4d17d89-b883-472d-84c9-b8f4d0c8eb7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf" Oct 30 17:52:10.504970 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.504938 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-tls\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.504970 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.504947 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8e818352-df23-46b4-84d9-81fd259200ec-config-volume\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:10.505057 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.504978 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-tls\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.505202 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:10.505187 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:52:10.505261 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:10.505207 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bf8b8bb7-dlchr: secret "image-registry-tls" not found Oct 30 17:52:10.505307 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:10.505297 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-tls podName:21a54420-0e75-4d4b-93cd-fe72c0cc2070 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:11.00527555 +0000 UTC m=+30.644608341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-tls") pod "image-registry-64bf8b8bb7-dlchr" (UID: "21a54420-0e75-4d4b-93cd-fe72c0cc2070") : secret "image-registry-tls" not found Oct 30 17:52:10.505360 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.505326 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbc66\" (UniqueName: \"kubernetes.io/projected/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e-kube-api-access-qbc66\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf\" (UID: \"b4d17d89-b883-472d-84c9-b8f4d0c8eb7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf" Oct 30 17:52:10.505402 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.505374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll92d\" (UniqueName: \"kubernetes.io/projected/d58a06a1-fdbc-439e-8572-fdeabc6ee4ef-kube-api-access-ll92d\") pod \"network-check-source-6f78d59f99-w725f\" (UID: \"d58a06a1-fdbc-439e-8572-fdeabc6ee4ef\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-w725f" Oct 30 17:52:10.505448 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.505407 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9xxt\" (UniqueName: \"kubernetes.io/projected/c9602b28-165a-4c17-96fa-9f0e25fd2cf8-kube-api-access-r9xxt\") pod \"ingress-canary-7txkr\" (UID: \"c9602b28-165a-4c17-96fa-9f0e25fd2cf8\") " pod="openshift-ingress-canary/ingress-canary-7txkr" Oct 30 17:52:10.505791 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.505768 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21a54420-0e75-4d4b-93cd-fe72c0cc2070-trusted-ca\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.506150 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.506127 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-ca-trust-extracted\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.506249 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:10.506235 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:52:10.506304 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:10.506253 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69885b9d-n2bmz: secret "image-registry-tls" not found Oct 30 17:52:10.506304 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:10.506300 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-tls podName:9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:11.00628613 +0000 UTC m=+30.645618919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-tls") pod "image-registry-69885b9d-n2bmz" (UID: "9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63") : secret "image-registry-tls" not found Oct 30 17:52:10.506821 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.506791 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21a54420-0e75-4d4b-93cd-fe72c0cc2070-ca-trust-extracted\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.507301 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.507281 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-certificates\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.507520 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:10.507487 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:52:10.507670 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:10.507659 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls podName:8e818352-df23-46b4-84d9-81fd259200ec nodeName:}" failed. No retries permitted until 2026-10-30 17:52:11.007641361 +0000 UTC m=+30.646974152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls") pod "dns-default-7r5h8" (UID: "8e818352-df23-46b4-84d9-81fd259200ec") : secret "dns-default-metrics-tls" not found Oct 30 17:52:10.508059 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.508039 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf\" (UID: \"b4d17d89-b883-472d-84c9-b8f4d0c8eb7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf" Oct 30 17:52:10.508428 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.508406 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf\" (UID: \"b4d17d89-b883-472d-84c9-b8f4d0c8eb7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf" Oct 30 17:52:10.509658 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.509637 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21a54420-0e75-4d4b-93cd-fe72c0cc2070-image-registry-private-configuration\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.510289 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.510008 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-installation-pull-secrets\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.510690 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.510305 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9602b28-165a-4c17-96fa-9f0e25fd2cf8-cert\") pod \"ingress-canary-7txkr\" (UID: \"c9602b28-165a-4c17-96fa-9f0e25fd2cf8\") " pod="openshift-ingress-canary/ingress-canary-7txkr" Oct 30 17:52:10.510805 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.510787 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-trusted-ca\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.511002 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.510965 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21a54420-0e75-4d4b-93cd-fe72c0cc2070-installation-pull-secrets\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.511071 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.511051 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-image-registry-private-configuration\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.533276 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.533255 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-bound-sa-token\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.537942 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.537923 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qplp\" (UniqueName: \"kubernetes.io/projected/8e818352-df23-46b4-84d9-81fd259200ec-kube-api-access-4qplp\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:10.544878 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.544851 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmwhl\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-kube-api-access-hmwhl\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.545104 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.545081 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgxhb\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-kube-api-access-vgxhb\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:10.545296 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.545272 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9xxt\" (UniqueName: \"kubernetes.io/projected/c9602b28-165a-4c17-96fa-9f0e25fd2cf8-kube-api-access-r9xxt\") pod \"ingress-canary-7txkr\" (UID: \"c9602b28-165a-4c17-96fa-9f0e25fd2cf8\") " pod="openshift-ingress-canary/ingress-canary-7txkr" Oct 30 17:52:10.545996 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.545965 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4bqp\" (UniqueName: \"kubernetes.io/projected/355e77f7-f944-4955-95c2-bdf4c7ea12b7-kube-api-access-d4bqp\") pod \"kas-connection-checker-685895c7c9-6xn6r\" (UID: \"355e77f7-f944-4955-95c2-bdf4c7ea12b7\") " pod="kube-system/kas-connection-checker-685895c7c9-6xn6r" Oct 30 17:52:10.546080 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.546040 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll92d\" (UniqueName: \"kubernetes.io/projected/d58a06a1-fdbc-439e-8572-fdeabc6ee4ef-kube-api-access-ll92d\") pod \"network-check-source-6f78d59f99-w725f\" (UID: \"d58a06a1-fdbc-439e-8572-fdeabc6ee4ef\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-w725f" Oct 30 17:52:10.546336 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.546316 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-bound-sa-token\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:10.547017 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.546997 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbc66\" (UniqueName: \"kubernetes.io/projected/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e-kube-api-access-qbc66\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf\" (UID: \"b4d17d89-b883-472d-84c9-b8f4d0c8eb7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf" Oct 30 17:52:10.553019 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.553003 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-w725f" Oct 30 17:52:10.564703 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.564685 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf" Oct 30 17:52:10.570825 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.570805 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:10.575623 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.575609 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6xn6r" Oct 30 17:52:10.619360 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.619341 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7txkr" Oct 30 17:52:10.754548 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.754520 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:10.911288 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.911264 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkgjk\" (UID: \"f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkgjk" Oct 30 17:52:10.911653 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:10.911376 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:52:10.911653 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:10.911424 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-networking-console-plugin-cert podName:f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:11.9114117 +0000 UTC m=+31.550744499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kkgjk" (UID: "f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2") : secret "networking-console-plugin-cert" not found Oct 30 17:52:10.946841 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.946785 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6xn6r"] Oct 30 17:52:10.952491 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:52:10.952471 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355e77f7_f944_4955_95c2_bdf4c7ea12b7.slice/crio-576a48cc07912595cac16d588f2857199ee4b0c8bc0b940d33575fe3873609c7 WatchSource:0}: Error finding container 576a48cc07912595cac16d588f2857199ee4b0c8bc0b940d33575fe3873609c7: Status 404 returned error can't find the container with id 576a48cc07912595cac16d588f2857199ee4b0c8bc0b940d33575fe3873609c7 Oct 30 17:52:10.958175 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.958146 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-w725f"] Oct 30 17:52:10.961897 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.961867 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7txkr"] Oct 30 17:52:10.961977 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:10.961923 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf"] Oct 30 17:52:10.965371 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:52:10.965348 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd58a06a1_fdbc_439e_8572_fdeabc6ee4ef.slice/crio-44ab6da95edbd8941f700dce2e52925fdd282921db1dab6c2230bc9d3203dbf4 WatchSource:0}: Error finding container 44ab6da95edbd8941f700dce2e52925fdd282921db1dab6c2230bc9d3203dbf4: Status 404 returned error can't find the container with id 44ab6da95edbd8941f700dce2e52925fdd282921db1dab6c2230bc9d3203dbf4 Oct 30 17:52:10.967365 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:52:10.967341 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d17d89_b883_472d_84c9_b8f4d0c8eb7e.slice/crio-88566190f852aad7ef96b6cdae8fa8b8c293ceb99d24ae02cfa03e50341a9400 WatchSource:0}: Error finding container 88566190f852aad7ef96b6cdae8fa8b8c293ceb99d24ae02cfa03e50341a9400: Status 404 returned error can't find the container with id 88566190f852aad7ef96b6cdae8fa8b8c293ceb99d24ae02cfa03e50341a9400 Oct 30 17:52:10.970946 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:52:10.967857 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9602b28_165a_4c17_96fa_9f0e25fd2cf8.slice/crio-038cb98c16682576ea3b21b367e72ebda6dfeafaf0e4643e705aff56bff3bebc WatchSource:0}: Error finding container 038cb98c16682576ea3b21b367e72ebda6dfeafaf0e4643e705aff56bff3bebc: Status 404 returned error can't find the container with id 038cb98c16682576ea3b21b367e72ebda6dfeafaf0e4643e705aff56bff3bebc Oct 30 17:52:11.012430 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:11.012406 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:11.012549 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:11.012443 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-tls\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:11.012549 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:11.012464 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-tls\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:11.012663 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:11.012578 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:52:11.012663 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:11.012591 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bf8b8bb7-dlchr: secret "image-registry-tls" not found Oct 30 17:52:11.012663 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:11.012635 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:52:11.012663 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:11.012642 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-tls podName:21a54420-0e75-4d4b-93cd-fe72c0cc2070 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:12.012626591 +0000 UTC m=+31.651959395 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-tls") pod "image-registry-64bf8b8bb7-dlchr" (UID: "21a54420-0e75-4d4b-93cd-fe72c0cc2070") : secret "image-registry-tls" not found Oct 30 17:52:11.012848 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:11.012667 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls podName:8e818352-df23-46b4-84d9-81fd259200ec nodeName:}" failed. No retries permitted until 2026-10-30 17:52:12.012659371 +0000 UTC m=+31.651992179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls") pod "dns-default-7r5h8" (UID: "8e818352-df23-46b4-84d9-81fd259200ec") : secret "dns-default-metrics-tls" not found Oct 30 17:52:11.012914 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:11.012896 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:52:11.012961 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:11.012922 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69885b9d-n2bmz: secret "image-registry-tls" not found Oct 30 17:52:11.013230 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:11.012974 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-tls podName:9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:12.012958538 +0000 UTC m=+31.652291344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-tls") pod "image-registry-69885b9d-n2bmz" (UID: "9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63") : secret "image-registry-tls" not found Oct 30 17:52:11.059455 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:11.059426 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:11.078662 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:11.078637 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6xn6r" event={"ID":"355e77f7-f944-4955-95c2-bdf4c7ea12b7","Type":"ContainerStarted","Data":"576a48cc07912595cac16d588f2857199ee4b0c8bc0b940d33575fe3873609c7"} Oct 30 17:52:11.079917 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:11.079892 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf" event={"ID":"b4d17d89-b883-472d-84c9-b8f4d0c8eb7e","Type":"ContainerStarted","Data":"88566190f852aad7ef96b6cdae8fa8b8c293ceb99d24ae02cfa03e50341a9400"} Oct 30 17:52:11.080999 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:11.080976 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-w725f" event={"ID":"d58a06a1-fdbc-439e-8572-fdeabc6ee4ef","Type":"ContainerStarted","Data":"44ab6da95edbd8941f700dce2e52925fdd282921db1dab6c2230bc9d3203dbf4"} Oct 30 17:52:11.082040 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:11.082021 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7txkr" event={"ID":"c9602b28-165a-4c17-96fa-9f0e25fd2cf8","Type":"ContainerStarted","Data":"038cb98c16682576ea3b21b367e72ebda6dfeafaf0e4643e705aff56bff3bebc"} Oct 30 17:52:11.777270 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:11.777237 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pgghg" Oct 30 17:52:11.777441 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:11.777380 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:52:11.778373 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:11.778353 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pgghg" Oct 30 17:52:11.828000 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:11.827975 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:52:11.828145 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:11.827980 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:52:11.828145 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:11.828124 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:52:11.840547 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:11.840526 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 17:52:11.840717 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:11.840671 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zlfnf\"" Oct 30 17:52:11.840828 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:11.840725 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ddhr5\"" Oct 30 17:52:11.841005 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:11.840988 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-x8qzn\"" Oct 30 17:52:11.841101 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:11.840989 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 17:52:11.920124 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:11.920090 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkgjk\" (UID: \"f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkgjk" Oct 30 17:52:11.920486 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:11.920279 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:52:11.920486 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:11.920396 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-networking-console-plugin-cert podName:f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:13.920375565 +0000 UTC m=+33.559708365 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kkgjk" (UID: "f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2") : secret "networking-console-plugin-cert" not found Oct 30 17:52:11.922123 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:11.922100 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:12.022125 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:12.022089 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:12.022281 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:12.022149 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-tls\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:12.022281 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:12.022202 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-tls\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:12.022390 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:12.022376 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:52:12.022433 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:12.022392 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bf8b8bb7-dlchr: secret "image-registry-tls" not found Oct 30 17:52:12.022477 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:12.022437 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-tls podName:21a54420-0e75-4d4b-93cd-fe72c0cc2070 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:14.022423898 +0000 UTC m=+33.661756704 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-tls") pod "image-registry-64bf8b8bb7-dlchr" (UID: "21a54420-0e75-4d4b-93cd-fe72c0cc2070") : secret "image-registry-tls" not found Oct 30 17:52:12.023028 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:12.023011 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:52:12.023072 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:12.023056 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls podName:8e818352-df23-46b4-84d9-81fd259200ec nodeName:}" failed. No retries permitted until 2026-10-30 17:52:14.023045967 +0000 UTC m=+33.662378765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls") pod "dns-default-7r5h8" (UID: "8e818352-df23-46b4-84d9-81fd259200ec") : secret "dns-default-metrics-tls" not found Oct 30 17:52:12.023115 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:12.023104 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:52:12.023147 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:12.023114 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69885b9d-n2bmz: secret "image-registry-tls" not found Oct 30 17:52:12.023147 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:12.023138 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-tls podName:9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:14.023129668 +0000 UTC m=+33.662462466 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-tls") pod "image-registry-69885b9d-n2bmz" (UID: "9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63") : secret "image-registry-tls" not found Oct 30 17:52:12.088616 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:12.088547 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6xn6r" event={"ID":"355e77f7-f944-4955-95c2-bdf4c7ea12b7","Type":"ContainerStarted","Data":"d10de38fa43451e4a24cf98fea114eb0561e77e1f636697dfb2c1270696285ce"} Oct 30 17:52:12.123740 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:12.123686 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6xn6r" podStartSLOduration=400.123671752 podStartE2EDuration="6m40.123671752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:12.123138701 +0000 UTC m=+31.762471509" watchObservedRunningTime="2026-10-30 17:52:12.123671752 +0000 UTC m=+31.763004589" Oct 30 17:52:13.361024 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:13.360971 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:13.505316 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:13.505283 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:13.637870 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:13.637791 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret\") pod \"global-pull-secret-syncer-5drgz\" (UID: \"1e6590e8-bfc4-4b19-b579-7180aa6af22a\") " pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:52:13.637870 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:13.637841 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs\") pod \"network-metrics-daemon-jnxk7\" (UID: \"827bbba0-b306-43db-b805-8c87dfabb874\") " pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:52:13.638115 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:13.638093 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:52:13.638259 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:13.638157 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs podName:827bbba0-b306-43db-b805-8c87dfabb874 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:45.638143762 +0000 UTC m=+65.277476573 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs") pod "network-metrics-daemon-jnxk7" (UID: "827bbba0-b306-43db-b805-8c87dfabb874") : secret "metrics-daemon-secret" not found Oct 30 17:52:13.640990 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:13.640969 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret\") pod \"global-pull-secret-syncer-5drgz\" (UID: \"1e6590e8-bfc4-4b19-b579-7180aa6af22a\") " pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:52:13.720632 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:13.720600 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:52:13.738875 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:13.738847 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbs6z\" (UniqueName: \"kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z\") pod \"network-check-target-4xxmr\" (UID: \"90437f91-1657-47db-b3a1-d59bc8d1de14\") " pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:52:13.741798 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:13.741770 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbs6z\" (UniqueName: \"kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z\") pod \"network-check-target-4xxmr\" (UID: \"90437f91-1657-47db-b3a1-d59bc8d1de14\") " pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:52:13.902308 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:13.902270 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:13.940551 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:13.940520 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkgjk\" (UID: \"f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkgjk" Oct 30 17:52:13.940703 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:13.940654 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:52:13.940744 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:13.940715 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-networking-console-plugin-cert podName:f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:17.940702252 +0000 UTC m=+37.580035039 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kkgjk" (UID: "f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2") : secret "networking-console-plugin-cert" not found Oct 30 17:52:13.998723 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:13.998695 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:52:14.041266 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:14.041243 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-tls\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:14.041364 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:14.041282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-tls\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:14.041425 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:14.041396 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:52:14.041425 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:14.041399 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:52:14.041425 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:14.041425 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69885b9d-n2bmz: secret "image-registry-tls" not found Oct 30 17:52:14.041685 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:14.041411 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bf8b8bb7-dlchr: secret "image-registry-tls" not found Oct 30 17:52:14.041685 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:14.041482 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-tls podName:9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:18.041463845 +0000 UTC m=+37.680796638 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-tls") pod "image-registry-69885b9d-n2bmz" (UID: "9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63") : secret "image-registry-tls" not found Oct 30 17:52:14.041685 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:14.041529 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-tls podName:21a54420-0e75-4d4b-93cd-fe72c0cc2070 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:18.04151413 +0000 UTC m=+37.680846931 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-tls") pod "image-registry-64bf8b8bb7-dlchr" (UID: "21a54420-0e75-4d4b-93cd-fe72c0cc2070") : secret "image-registry-tls" not found Oct 30 17:52:14.041685 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:14.041574 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:14.041685 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:14.041671 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:52:14.041879 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:14.041709 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls podName:8e818352-df23-46b4-84d9-81fd259200ec nodeName:}" failed. No retries permitted until 2026-10-30 17:52:18.041697572 +0000 UTC m=+37.681030360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls") pod "dns-default-7r5h8" (UID: "8e818352-df23-46b4-84d9-81fd259200ec") : secret "dns-default-metrics-tls" not found Oct 30 17:52:14.055754 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:14.055717 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5drgz"] Oct 30 17:52:14.096855 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:14.096826 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf" event={"ID":"b4d17d89-b883-472d-84c9-b8f4d0c8eb7e","Type":"ContainerStarted","Data":"3da433ccc715ed4d16a8024ef01191d057f3872bc6bb7dc9157724e6031ed464"} Oct 30 17:52:14.098644 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:14.098617 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5drgz" event={"ID":"1e6590e8-bfc4-4b19-b579-7180aa6af22a","Type":"ContainerStarted","Data":"f2ac51125accf1e753569ce7a42fd3450827e17dc4c78b239189f6d78ae5973c"} Oct 30 17:52:14.135620 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:14.135572 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf" podStartSLOduration=366.337047453 podStartE2EDuration="6m9.13555952s" totalImagesPullingTime="2.798512067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:14.135030454 +0000 UTC m=+33.774363289" watchObservedRunningTime="2026-10-30 17:52:14.13555952 +0000 UTC m=+33.774892322" Oct 30 17:52:14.326474 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:14.326450 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4xxmr"] Oct 30 17:52:14.332107 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:52:14.332079 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90437f91_1657_47db_b3a1_d59bc8d1de14.slice/crio-469f74b30406a1a093d82860170c4647c39ad02766381d22db7a00389fa9ac0b WatchSource:0}: Error finding container 469f74b30406a1a093d82860170c4647c39ad02766381d22db7a00389fa9ac0b: Status 404 returned error can't find the container with id 469f74b30406a1a093d82860170c4647c39ad02766381d22db7a00389fa9ac0b Oct 30 17:52:14.750065 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:14.750030 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jr2r5"] Oct 30 17:52:14.755106 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:14.755081 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:14.768211 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:14.768190 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jr2r5" Oct 30 17:52:14.780927 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:14.780904 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jr2r5"] Oct 30 17:52:14.784346 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:14.784324 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-cgqqv\"" Oct 30 17:52:14.788216 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:14.788165 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 17:52:14.788303 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:14.788254 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 17:52:14.935417 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:14.935369 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:14.948863 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:14.948833 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z85fd\" (UniqueName: \"kubernetes.io/projected/1a6fe435-1778-45b0-ae18-6695f5824229-kube-api-access-z85fd\") pod \"migrator-66df5d496b-jr2r5\" (UID: \"1a6fe435-1778-45b0-ae18-6695f5824229\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jr2r5" Oct 30 17:52:15.049844 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.049776 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z85fd\" (UniqueName: \"kubernetes.io/projected/1a6fe435-1778-45b0-ae18-6695f5824229-kube-api-access-z85fd\") pod \"migrator-66df5d496b-jr2r5\" (UID: \"1a6fe435-1778-45b0-ae18-6695f5824229\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jr2r5" Oct 30 17:52:15.073675 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.073652 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z85fd\" (UniqueName: \"kubernetes.io/projected/1a6fe435-1778-45b0-ae18-6695f5824229-kube-api-access-z85fd\") pod \"migrator-66df5d496b-jr2r5\" (UID: \"1a6fe435-1778-45b0-ae18-6695f5824229\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jr2r5" Oct 30 17:52:15.097417 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.097394 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jr2r5" Oct 30 17:52:15.104163 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.103764 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-w725f" event={"ID":"d58a06a1-fdbc-439e-8572-fdeabc6ee4ef","Type":"ContainerStarted","Data":"92e41cc2a4da304d2b04ff115317a74fb605da737e5e44f2ce868bca384f20a9"} Oct 30 17:52:15.106154 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.106129 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7txkr" event={"ID":"c9602b28-165a-4c17-96fa-9f0e25fd2cf8","Type":"ContainerStarted","Data":"ccec1c1337571a9e086e4e65c2c49f20196e6ea6694dae3d95314459cfa537c4"} Oct 30 17:52:15.107922 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.107894 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4xxmr" event={"ID":"90437f91-1657-47db-b3a1-d59bc8d1de14","Type":"ContainerStarted","Data":"75b179d4ed4c3fa6df7f9cfcd35f53ea7cda581cc165a74ee109642b41540c81"} Oct 30 17:52:15.108034 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.107941 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4xxmr" event={"ID":"90437f91-1657-47db-b3a1-d59bc8d1de14","Type":"ContainerStarted","Data":"469f74b30406a1a093d82860170c4647c39ad02766381d22db7a00389fa9ac0b"} Oct 30 17:52:15.108034 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.108010 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:52:15.115884 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.115862 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:15.136793 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.136765 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nsw4r"] Oct 30 17:52:15.155746 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.155717 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:15.179636 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.173242 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-w725f" podStartSLOduration=361.602018672 podStartE2EDuration="6m5.173221776s" totalImagesPullingTime="3.571203104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:15.154437138 +0000 UTC m=+34.793769947" watchObservedRunningTime="2026-10-30 17:52:15.173221776 +0000 UTC m=+34.812554586" Oct 30 17:52:15.179636 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.174158 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nsw4r"] Oct 30 17:52:15.201910 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.200567 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4xxmr" podStartSLOduration=33.754197841 podStartE2EDuration="34.20054864s" totalImagesPullingTime="446.350799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:15.1991008 +0000 UTC m=+34.838433610" watchObservedRunningTime="2026-10-30 17:52:15.20054864 +0000 UTC m=+34.839881451" Oct 30 17:52:15.239738 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.239698 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 17:52:15.239941 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.239907 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 17:52:15.240069 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.239909 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 17:52:15.245772 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.245713 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7txkr" podStartSLOduration=1.918573316 podStartE2EDuration="5.245696002s" totalImagesPullingTime="3.327122686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:15.245194068 +0000 UTC m=+34.884526875" watchObservedRunningTime="2026-10-30 17:52:15.245696002 +0000 UTC m=+34.885028811" Oct 30 17:52:15.251833 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.251804 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:15.251964 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.251846 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72993a50-bf5d-40c5-833b-afff9f86e61d-data-volume\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:15.251964 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.251873 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72993a50-bf5d-40c5-833b-afff9f86e61d-crio-socket\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:15.252349 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.252025 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxtq4\" (UniqueName: \"kubernetes.io/projected/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-api-access-lxtq4\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:15.252349 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.252064 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:15.266263 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.266241 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 17:52:15.266604 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.266578 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cxbhw\"" Oct 30 17:52:15.291548 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.291524 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:15.353256 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.353180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:15.353256 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.353248 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:15.353428 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.353273 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72993a50-bf5d-40c5-833b-afff9f86e61d-data-volume\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:15.353428 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.353292 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72993a50-bf5d-40c5-833b-afff9f86e61d-crio-socket\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:15.353428 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:15.353354 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:52:15.353428 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.353375 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxtq4\" (UniqueName: \"kubernetes.io/projected/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-api-access-lxtq4\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:15.353428 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:15.353387 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:52:15.353428 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:15.353430 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls podName:72993a50-bf5d-40c5-833b-afff9f86e61d nodeName:}" failed. No retries permitted until 2026-10-30 17:52:15.853407166 +0000 UTC m=+35.492739955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nsw4r" (UID: "72993a50-bf5d-40c5-833b-afff9f86e61d") : secret "insights-runtime-extractor-tls" not found Oct 30 17:52:15.353734 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:15.353453 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-rbac-proxy-cm podName:72993a50-bf5d-40c5-833b-afff9f86e61d nodeName:}" failed. No retries permitted until 2026-10-30 17:52:15.853438397 +0000 UTC m=+35.492771207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nsw4r" (UID: "72993a50-bf5d-40c5-833b-afff9f86e61d") : configmap "kube-rbac-proxy" not found Oct 30 17:52:15.353779 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.353735 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72993a50-bf5d-40c5-833b-afff9f86e61d-crio-socket\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:15.353895 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.353877 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72993a50-bf5d-40c5-833b-afff9f86e61d-data-volume\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:15.379301 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.379276 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxtq4\" (UniqueName: \"kubernetes.io/projected/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-api-access-lxtq4\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:15.440709 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.440675 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jr2r5"] Oct 30 17:52:15.445678 ip-10-0-128-236 kubenswrapper[2578]: W1030 17:52:15.445650 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6fe435_1778_45b0_ae18_6695f5824229.slice/crio-cf000717c4c846facb91c7480cc7c5102ec411c4615ce12d44b92cccb8724127 WatchSource:0}: Error finding container cf000717c4c846facb91c7480cc7c5102ec411c4615ce12d44b92cccb8724127: Status 404 returned error can't find the container with id cf000717c4c846facb91c7480cc7c5102ec411c4615ce12d44b92cccb8724127 Oct 30 17:52:15.564388 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.564357 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:15.719371 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.719268 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:15.857673 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.857640 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:15.858138 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:15.857738 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:15.858138 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:15.857853 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:52:15.858138 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:15.857893 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:52:15.858138 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:15.857917 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-rbac-proxy-cm podName:72993a50-bf5d-40c5-833b-afff9f86e61d nodeName:}" failed. No retries permitted until 2026-10-30 17:52:16.857900401 +0000 UTC m=+36.497233190 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nsw4r" (UID: "72993a50-bf5d-40c5-833b-afff9f86e61d") : configmap "kube-rbac-proxy" not found Oct 30 17:52:15.858138 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:15.857956 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls podName:72993a50-bf5d-40c5-833b-afff9f86e61d nodeName:}" failed. No retries permitted until 2026-10-30 17:52:16.857939239 +0000 UTC m=+36.497272042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nsw4r" (UID: "72993a50-bf5d-40c5-833b-afff9f86e61d") : secret "insights-runtime-extractor-tls" not found Oct 30 17:52:16.112764 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:16.112725 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jr2r5" event={"ID":"1a6fe435-1778-45b0-ae18-6695f5824229","Type":"ContainerStarted","Data":"cf000717c4c846facb91c7480cc7c5102ec411c4615ce12d44b92cccb8724127"} Oct 30 17:52:16.668350 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:16.668304 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:16.812112 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:16.812073 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:16.865667 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:16.865638 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:16.866107 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:16.865706 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:16.866107 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:16.865799 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:52:16.866107 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:16.865836 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:52:16.866107 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:16.865863 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls podName:72993a50-bf5d-40c5-833b-afff9f86e61d nodeName:}" failed. No retries permitted until 2026-10-30 17:52:18.865843951 +0000 UTC m=+38.505176738 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nsw4r" (UID: "72993a50-bf5d-40c5-833b-afff9f86e61d") : secret "insights-runtime-extractor-tls" not found Oct 30 17:52:16.866107 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:16.865895 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-rbac-proxy-cm podName:72993a50-bf5d-40c5-833b-afff9f86e61d nodeName:}" failed. No retries permitted until 2026-10-30 17:52:18.86587838 +0000 UTC m=+38.505211183 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nsw4r" (UID: "72993a50-bf5d-40c5-833b-afff9f86e61d") : configmap "kube-rbac-proxy" not found Oct 30 17:52:17.117240 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:17.117205 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jr2r5" event={"ID":"1a6fe435-1778-45b0-ae18-6695f5824229","Type":"ContainerStarted","Data":"41226918dc6e2e13ccf8f8031eb47c7ce4ff26b12e46a1ddc414da61395367d6"} Oct 30 17:52:17.158807 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:17.158737 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jr2r5" podStartSLOduration=1.6002121900000001 podStartE2EDuration="3.158719034s" totalImagesPullingTime="1.558506844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:17.158302879 +0000 UTC m=+36.797635688" watchObservedRunningTime="2026-10-30 17:52:17.158719034 +0000 UTC m=+36.798051867" Oct 30 17:52:17.566382 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:17.566342 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:17.975640 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:17.975550 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkgjk\" (UID: \"f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkgjk" Oct 30 17:52:17.976081 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:17.975750 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:52:17.976081 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:17.975824 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-networking-console-plugin-cert podName:f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:25.975803102 +0000 UTC m=+45.615135890 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kkgjk" (UID: "f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2") : secret "networking-console-plugin-cert" not found Oct 30 17:52:18.076275 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:18.076239 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:18.076458 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:18.076293 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-tls\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:18.076458 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:18.076348 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-tls\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:18.076458 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:18.076413 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:52:18.076458 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:18.076450 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:52:18.076641 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:18.076464 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bf8b8bb7-dlchr: secret "image-registry-tls" not found Oct 30 17:52:18.076641 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:18.076484 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls podName:8e818352-df23-46b4-84d9-81fd259200ec nodeName:}" failed. No retries permitted until 2026-10-30 17:52:26.076465432 +0000 UTC m=+45.715798230 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls") pod "dns-default-7r5h8" (UID: "8e818352-df23-46b4-84d9-81fd259200ec") : secret "dns-default-metrics-tls" not found Oct 30 17:52:18.076641 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:18.076530 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-tls podName:21a54420-0e75-4d4b-93cd-fe72c0cc2070 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:26.076514122 +0000 UTC m=+45.715846921 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-tls") pod "image-registry-64bf8b8bb7-dlchr" (UID: "21a54420-0e75-4d4b-93cd-fe72c0cc2070") : secret "image-registry-tls" not found Oct 30 17:52:18.076641 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:18.076606 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:52:18.076641 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:18.076622 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69885b9d-n2bmz: secret "image-registry-tls" not found Oct 30 17:52:18.076837 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:18.076656 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-tls podName:9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:26.076645325 +0000 UTC m=+45.715978124 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-tls") pod "image-registry-69885b9d-n2bmz" (UID: "9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63") : secret "image-registry-tls" not found Oct 30 17:52:18.136185 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:18.136151 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:18.306953 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:18.306923 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:18.518988 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:18.518944 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:18.728768 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:18.728699 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:18.883970 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:18.883936 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:18.884105 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:18.884003 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:18.884105 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:18.884083 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:52:18.884105 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:18.884100 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:52:18.884215 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:18.884146 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-rbac-proxy-cm podName:72993a50-bf5d-40c5-833b-afff9f86e61d nodeName:}" failed. No retries permitted until 2026-10-30 17:52:22.884135293 +0000 UTC m=+42.523468091 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nsw4r" (UID: "72993a50-bf5d-40c5-833b-afff9f86e61d") : configmap "kube-rbac-proxy" not found Oct 30 17:52:18.884215 ip-10-0-128-236 kubenswrapper[2578]: E1030 17:52:18.884159 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls podName:72993a50-bf5d-40c5-833b-afff9f86e61d nodeName:}" failed. No retries permitted until 2026-10-30 17:52:22.884153194 +0000 UTC m=+42.523485980 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nsw4r" (UID: "72993a50-bf5d-40c5-833b-afff9f86e61d") : secret "insights-runtime-extractor-tls" not found Oct 30 17:52:18.970702 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:18.970665 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:19.125355 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:19.125327 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5drgz" event={"ID":"1e6590e8-bfc4-4b19-b579-7180aa6af22a","Type":"ContainerStarted","Data":"fb38e9494666decdc9ec9eeafb54eb16133564476a1a08fdab92ea20b0bebd88"} Oct 30 17:52:19.161080 ip-10-0-128-236 kubenswrapper[2578]: I1030 17:52:19.161036 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5drgz" podStartSLOduration=33.293764447 podStartE2EDuration="38.1610219s" totalImagesPullingTime="4.867257453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:19.159827874 +0000 UTC m=+38.799160681" watchObservedRunningTime="2026-10-30 17:52:19.1610219 +0000 UTC m=+38.800354704" Oct 30 17:52:19.210974 ip-10-0-128-236 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 17:52:19.234845 ip-10-0-128-236 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 17:52:19.235063 ip-10-0-128-236 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 17:52:19.235124 ip-10-0-128-236 systemd[1]: kubelet.service: Consumed 2.480s CPU time. Oct 30 17:52:19.243066 ip-10-0-128-236 systemd[1]: Starting Kubernetes Kubelet... Oct 30 17:52:19.332084 ip-10-0-128-236 kubenswrapper[4603]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:52:19.332084 ip-10-0-128-236 kubenswrapper[4603]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 17:52:19.332084 ip-10-0-128-236 kubenswrapper[4603]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:52:19.332084 ip-10-0-128-236 kubenswrapper[4603]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:52:19.334146 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334077 4603 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:19.334146 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334145 4603 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:19.334146 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334148 4603 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:19.334238 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334152 4603 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:19.334238 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334156 4603 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:19.334238 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334159 4603 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:19.334238 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334162 4603 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:19.334238 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334165 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:19.334238 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334168 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:19.334238 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334170 4603 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:19.334238 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334173 4603 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:19.334238 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334176 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:19.334238 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334178 4603 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:19.334238 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334181 4603 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:19.334238 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334183 4603 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:19.334238 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334186 4603 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:19.334238 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334189 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:19.334238 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334191 4603 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:19.334238 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334194 4603 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:19.334238 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334197 4603 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:19.334238 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334200 4603 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:19.334238 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334203 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:19.334735 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334209 4603 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:19.334735 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334213 4603 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:19.334735 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334216 4603 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:19.334735 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334218 4603 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:19.334735 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334221 4603 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:19.334735 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334224 4603 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:19.334735 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334227 4603 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:19.334735 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334229 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:19.334735 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334232 4603 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:19.334735 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334235 4603 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:19.334735 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334237 4603 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:19.334735 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334240 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:19.334735 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334243 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:19.334735 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334246 4603 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:19.334735 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334249 4603 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:19.334735 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334251 4603 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:19.334735 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334254 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:19.334735 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334257 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:19.334735 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334260 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:19.335208 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334263 4603 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:19.335208 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334265 4603 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:19.335208 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334268 4603 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:19.335208 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334271 4603 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:19.335208 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334273 4603 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:19.335208 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334276 4603 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:19.335208 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334279 4603 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:19.335208 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334281 4603 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:19.335208 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334284 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:19.335208 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334287 4603 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:19.335208 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334290 4603 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:19.335208 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334293 4603 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:19.335208 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334295 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:19.335208 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334298 4603 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:19.335208 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334301 4603 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:19.335208 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334304 4603 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:19.335208 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334307 4603 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:19.335208 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334309 4603 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:19.335208 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334312 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:19.335208 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334315 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:19.335708 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334317 4603 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:19.335708 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334320 4603 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:19.335708 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334322 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:19.335708 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334325 4603 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:19.335708 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334328 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:19.335708 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334330 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:19.335708 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334333 4603 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:19.335708 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334336 4603 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:19.335708 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334339 4603 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:19.335708 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334341 4603 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:19.335708 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334344 4603 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:19.335708 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334348 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:19.335708 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334351 4603 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:19.335708 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334354 4603 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:19.335708 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334356 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:19.335708 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334359 4603 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:19.335708 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334361 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:19.335708 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334364 4603 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:19.335708 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334367 4603 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:19.335708 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334370 4603 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:19.336182 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334372 4603 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:19.336182 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334375 4603 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:19.336182 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334378 4603 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:19.336182 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334380 4603 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:19.336182 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334383 4603 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:19.336182 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334386 4603 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:19.336182 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334389 4603 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:19.336182 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334392 4603 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:19.336182 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334394 4603 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:19.336182 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334397 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:19.336182 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334399 4603 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:19.336182 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334402 4603 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:19.336182 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334404 4603 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:19.336182 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334407 4603 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:19.336182 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334410 4603 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:19.336182 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334414 4603 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:19.336182 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334418 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:19.336182 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334421 4603 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:19.336182 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334424 4603 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:19.336182 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334427 4603 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:19.336783 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334430 4603 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:19.336783 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334432 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:19.336783 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334435 4603 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:19.336783 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334438 4603 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:19.336783 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.334446 4603 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:19.336783 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335222 4603 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:19.336783 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335229 4603 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:19.336783 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335232 4603 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:19.336783 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335235 4603 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:19.336783 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335238 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:19.336783 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335241 4603 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:19.336783 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335244 4603 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:19.336783 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335247 4603 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:19.336783 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335250 4603 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:19.336783 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335254 4603 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:19.336783 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335257 4603 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:19.336783 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335260 4603 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:19.336783 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335263 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:19.336783 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335266 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:19.337262 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335269 4603 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:19.337262 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335271 4603 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:19.337262 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335274 4603 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:19.337262 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335277 4603 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:19.337262 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335280 4603 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:19.337262 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335282 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:19.337262 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335285 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:19.337262 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335287 4603 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:19.337262 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335290 4603 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:19.337262 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335293 4603 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:19.337262 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335296 4603 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:19.337262 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335298 4603 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:19.337262 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335301 4603 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:19.337262 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335304 4603 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:19.337262 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335306 4603 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:19.337262 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335309 4603 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:19.337262 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335311 4603 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:19.337262 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335314 4603 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:19.337262 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335318 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:19.337262 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335320 4603 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:19.337797 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335323 4603 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:19.337797 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335325 4603 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:19.337797 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335328 4603 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:19.337797 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335331 4603 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:19.337797 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335333 4603 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:19.337797 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335336 4603 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:19.337797 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335338 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:19.337797 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335341 4603 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:19.337797 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335343 4603 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:19.337797 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335346 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:19.337797 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335348 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:19.337797 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335351 4603 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:19.337797 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335354 4603 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:19.337797 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335356 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:19.337797 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335359 4603 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:19.337797 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335362 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:19.337797 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335365 4603 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:19.337797 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335367 4603 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:19.337797 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335370 4603 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:19.337797 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335372 4603 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:19.338288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335375 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:19.338288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335377 4603 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:19.338288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335380 4603 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:19.338288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335383 4603 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:19.338288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335385 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:19.338288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335388 4603 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:19.338288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335390 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:19.338288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335393 4603 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:19.338288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335396 4603 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:19.338288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335398 4603 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:19.338288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335401 4603 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:19.338288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335404 4603 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:19.338288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335407 4603 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:19.338288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335409 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:19.338288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335412 4603 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:19.338288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335414 4603 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:19.338288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335417 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:19.338288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335420 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:19.338288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335422 4603 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:19.338771 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335424 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:19.338771 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335427 4603 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:19.338771 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335429 4603 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:19.338771 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335432 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:19.338771 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335434 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:19.338771 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335437 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:19.338771 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335440 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:19.338771 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335442 4603 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:19.338771 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335445 4603 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:19.338771 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335447 4603 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:19.338771 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335450 4603 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:19.338771 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335452 4603 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:19.338771 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335454 4603 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:19.338771 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335457 4603 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:19.338771 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335460 4603 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:19.338771 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335462 4603 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:19.338771 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335465 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:19.338771 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335477 4603 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:19.338771 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335480 4603 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:19.338771 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335482 4603 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:19.339288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335485 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:19.339288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335488 4603 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:19.339288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335492 4603 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:19.339288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335511 4603 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:19.339288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335515 4603 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:19.339288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335518 4603 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:19.339288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335521 4603 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:19.339288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335523 4603 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:19.339288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335527 4603 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:19.339288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335529 4603 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:19.339288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335532 4603 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:19.339288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335535 4603 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:19.339288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335537 4603 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:19.339288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335993 4603 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:19.339288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.335998 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:19.339288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336001 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:19.339288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336004 4603 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:19.339288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336007 4603 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:19.339288 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336009 4603 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:19.339768 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336012 4603 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:19.339768 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336014 4603 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:19.339768 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336017 4603 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:19.339768 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336020 4603 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:19.339768 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336023 4603 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:19.339768 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336025 4603 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:19.339768 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336028 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:19.339768 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336031 4603 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:19.339768 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336039 4603 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:19.339768 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336043 4603 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:19.339768 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336046 4603 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:19.339768 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336049 4603 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:19.339768 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336052 4603 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:19.339768 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336054 4603 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:19.339768 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336059 4603 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:19.339768 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336062 4603 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:19.339768 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336065 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:19.339768 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336067 4603 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:19.339768 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336070 4603 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:19.339768 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336073 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:19.340278 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336076 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:19.340278 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336078 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:19.340278 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336081 4603 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:19.340278 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336083 4603 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:19.340278 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336086 4603 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:19.340278 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336089 4603 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:19.340278 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336091 4603 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:19.340278 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336094 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:19.340278 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336097 4603 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:19.340278 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336099 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:19.340278 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336102 4603 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:19.340278 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336105 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:19.340278 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336107 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:19.340278 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336111 4603 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:19.340278 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336114 4603 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:19.340278 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336117 4603 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:19.340278 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336121 4603 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:19.340278 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336123 4603 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:19.340278 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336126 4603 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:19.340924 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336128 4603 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:19.340924 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336134 4603 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:19.340924 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336136 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:19.340924 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336139 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:19.340924 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336142 4603 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:19.340924 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336144 4603 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:19.340924 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336147 4603 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:19.340924 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336154 4603 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:19.340924 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336157 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:19.340924 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336160 4603 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:19.340924 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336162 4603 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:19.340924 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336165 4603 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:19.340924 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336168 4603 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:19.340924 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336171 4603 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:19.340924 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336174 4603 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:19.340924 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336177 4603 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:19.340924 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336180 4603 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:19.340924 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336183 4603 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:19.340924 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336185 4603 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:19.341386 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336188 4603 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:19.341386 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336191 4603 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:19.341386 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336193 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:19.341386 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336196 4603 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:19.341386 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336198 4603 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:19.341386 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336201 4603 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:19.341386 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336204 4603 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:19.341386 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336206 4603 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:19.341386 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336208 4603 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:19.341386 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336211 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:19.341386 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336213 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:19.341386 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336216 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:19.341386 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336219 4603 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:19.341386 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336221 4603 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:19.341386 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336225 4603 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:19.341386 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336228 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:19.341386 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336230 4603 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:19.341386 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336233 4603 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:19.341386 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336235 4603 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:19.341863 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336238 4603 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:19.341863 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336242 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:19.341863 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336245 4603 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:19.341863 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336247 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:19.341863 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336250 4603 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:19.341863 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336252 4603 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:19.341863 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336255 4603 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:19.341863 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336258 4603 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:19.341863 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336260 4603 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:19.341863 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336263 4603 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:19.341863 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336266 4603 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:19.341863 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336268 4603 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:19.341863 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336271 4603 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:19.341863 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336273 4603 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:19.341863 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336276 4603 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:19.341863 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336278 4603 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:19.341863 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336281 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:19.341863 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336283 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:19.341863 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336286 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:19.341863 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336289 4603 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336292 4603 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336295 4603 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.336297 4603 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336387 4603 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336398 4603 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336403 4603 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336408 4603 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336413 4603 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336417 4603 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336421 4603 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336426 4603 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336429 4603 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336432 4603 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336437 4603 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336440 4603 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336443 4603 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336447 4603 flags.go:64] FLAG: --cgroup-root="" Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336449 4603 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336452 4603 flags.go:64] FLAG: --client-ca-file="" Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336455 4603 flags.go:64] FLAG: --cloud-provider="external" Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336459 4603 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336463 4603 flags.go:64] FLAG: --cluster-domain="" Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336466 4603 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 17:52:19.342377 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336469 4603 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336473 4603 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336476 4603 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336480 4603 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336483 4603 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336486 4603 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336490 4603 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336493 4603 flags.go:64] FLAG: --contention-profiling="false" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336496 4603 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336512 4603 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336515 4603 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336518 4603 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336523 4603 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336526 4603 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336529 4603 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336532 4603 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336537 4603 flags.go:64] FLAG: --enable-server="true" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336540 4603 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336545 4603 flags.go:64] FLAG: --event-burst="100" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336548 4603 flags.go:64] FLAG: --event-qps="50" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336551 4603 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336554 4603 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336558 4603 flags.go:64] FLAG: --eviction-hard="" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336562 4603 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 17:52:19.342979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336565 4603 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 17:52:19.343582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336568 4603 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 17:52:19.343582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336571 4603 flags.go:64] FLAG: --eviction-soft="" Oct 30 17:52:19.343582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336575 4603 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 17:52:19.343582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336577 4603 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 17:52:19.343582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336581 4603 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 17:52:19.343582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336584 4603 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 17:52:19.343582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336587 4603 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 17:52:19.343582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336590 4603 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 17:52:19.343582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336593 4603 flags.go:64] FLAG: --feature-gates="" Oct 30 17:52:19.343582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336597 4603 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 17:52:19.343582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336600 4603 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 17:52:19.343582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336603 4603 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 17:52:19.343582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336606 4603 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 17:52:19.343582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336609 4603 flags.go:64] FLAG: --healthz-port="10248" Oct 30 17:52:19.343582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336612 4603 flags.go:64] FLAG: --help="false" Oct 30 17:52:19.343582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336615 4603 flags.go:64] FLAG: --hostname-override="ip-10-0-128-236.ec2.internal" Oct 30 17:52:19.343582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336618 4603 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 17:52:19.343582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336621 4603 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 17:52:19.343582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336625 4603 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 17:52:19.343582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336628 4603 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 17:52:19.343582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336632 4603 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 17:52:19.343582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336635 4603 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 17:52:19.343582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336637 4603 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336642 4603 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336645 4603 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336648 4603 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336651 4603 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336654 4603 flags.go:64] FLAG: --kube-reserved="" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336657 4603 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336661 4603 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336664 4603 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336667 4603 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336670 4603 flags.go:64] FLAG: --lock-file="" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336673 4603 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336676 4603 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336679 4603 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336684 4603 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336687 4603 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336691 4603 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336694 4603 flags.go:64] FLAG: --logging-format="text" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336697 4603 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336701 4603 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336703 4603 flags.go:64] FLAG: --manifest-url="" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336706 4603 flags.go:64] FLAG: --manifest-url-header="" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336711 4603 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336713 4603 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336717 4603 flags.go:64] FLAG: --max-pods="110" Oct 30 17:52:19.344376 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336720 4603 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336723 4603 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336726 4603 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336729 4603 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336732 4603 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336735 4603 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336738 4603 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336745 4603 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336749 4603 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336752 4603 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336755 4603 flags.go:64] FLAG: --pod-cidr="" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336758 4603 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336761 4603 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336764 4603 flags.go:64] FLAG: --pods-per-core="0" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336768 4603 flags.go:64] FLAG: --port="10250" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336771 4603 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336774 4603 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c1b882d1b325481e" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336777 4603 flags.go:64] FLAG: --qos-reserved="" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336781 4603 flags.go:64] FLAG: --read-only-port="10255" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336784 4603 flags.go:64] FLAG: --register-node="true" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336787 4603 flags.go:64] FLAG: --register-with-taints="" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336791 4603 flags.go:64] FLAG: --registry-burst="10" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336794 4603 flags.go:64] FLAG: --registry-qps="5" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336797 4603 flags.go:64] FLAG: --reserved-cpus="" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336800 4603 flags.go:64] FLAG: --reserved-memory="" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336804 4603 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 17:52:19.345379 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336807 4603 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336810 4603 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336813 4603 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336816 4603 flags.go:64] FLAG: --runonce="false" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336819 4603 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336822 4603 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336825 4603 flags.go:64] FLAG: --seccomp-default="false" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336828 4603 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336831 4603 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336834 4603 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336837 4603 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336840 4603 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336843 4603 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336846 4603 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336850 4603 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336853 4603 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336856 4603 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336860 4603 flags.go:64] FLAG: --system-cgroups="" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336862 4603 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336868 4603 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336872 4603 flags.go:64] FLAG: --tls-cert-file="" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336875 4603 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336879 4603 flags.go:64] FLAG: --tls-min-version="" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336882 4603 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336885 4603 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 17:52:19.346047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336888 4603 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 17:52:19.346698 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336892 4603 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 17:52:19.346698 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336895 4603 flags.go:64] FLAG: --v="2" Oct 30 17:52:19.346698 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336899 4603 flags.go:64] FLAG: --version="false" Oct 30 17:52:19.346698 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336903 4603 flags.go:64] FLAG: --vmodule="" Oct 30 17:52:19.346698 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336907 4603 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 17:52:19.346698 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.336911 4603 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 17:52:19.346698 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337049 4603 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:19.346698 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337054 4603 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:19.346698 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337058 4603 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:19.346698 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337061 4603 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:19.346698 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337064 4603 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:19.346698 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337067 4603 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:19.346698 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337069 4603 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:19.346698 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337072 4603 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:19.346698 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337075 4603 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:19.346698 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337077 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:19.346698 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337080 4603 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:19.346698 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337083 4603 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:19.346698 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337086 4603 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:19.346698 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337088 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:19.346698 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337092 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:19.347291 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337095 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:19.347291 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337098 4603 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:19.347291 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337101 4603 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:19.347291 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337105 4603 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:19.347291 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337109 4603 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:19.347291 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337114 4603 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:19.347291 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337116 4603 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:19.347291 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337119 4603 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:19.347291 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337122 4603 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:19.347291 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337125 4603 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:19.347291 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337127 4603 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:19.347291 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337130 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:19.347291 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337133 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:19.347291 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337135 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:19.347291 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337138 4603 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:19.347291 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337140 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:19.347291 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337143 4603 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:19.347291 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337146 4603 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:19.347291 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337149 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:19.348009 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337152 4603 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:19.348009 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337154 4603 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:19.348009 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337157 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:19.348009 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337159 4603 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:19.348009 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337162 4603 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:19.348009 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337164 4603 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:19.348009 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337167 4603 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:19.348009 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337169 4603 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:19.348009 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337172 4603 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:19.348009 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337174 4603 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:19.348009 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337177 4603 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:19.348009 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337179 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:19.348009 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337183 4603 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:19.348009 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337186 4603 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:19.348009 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337190 4603 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:19.348009 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337193 4603 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:19.348009 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337196 4603 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:19.348009 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337199 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:19.348009 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337203 4603 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:19.348691 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337205 4603 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:19.348691 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337208 4603 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:19.348691 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337211 4603 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:19.348691 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337213 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:19.348691 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337216 4603 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:19.348691 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337218 4603 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:19.348691 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337221 4603 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:19.348691 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337224 4603 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:19.348691 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337226 4603 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:19.348691 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337229 4603 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:19.348691 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337231 4603 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:19.348691 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337234 4603 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:19.348691 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337237 4603 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:19.348691 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337240 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:19.348691 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337243 4603 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:19.348691 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337245 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:19.348691 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337248 4603 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:19.348691 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337250 4603 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:19.348691 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337253 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:19.348691 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337255 4603 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:19.349198 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337258 4603 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:19.349198 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337260 4603 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:19.349198 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337263 4603 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:19.349198 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337266 4603 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:19.349198 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337268 4603 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:19.349198 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337272 4603 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:19.349198 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337275 4603 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:19.349198 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337277 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:19.349198 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337280 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:19.349198 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337282 4603 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:19.349198 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337285 4603 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:19.349198 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337288 4603 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:19.349198 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337291 4603 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:19.349198 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337293 4603 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:19.349198 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337296 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:19.349198 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337298 4603 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:19.349198 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337301 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:19.349198 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337304 4603 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:19.349198 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337306 4603 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:19.349198 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337309 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:19.349928 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337311 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:19.349928 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337314 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:19.349928 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337317 4603 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:19.349928 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337319 4603 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:19.349928 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337322 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:19.349928 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337325 4603 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:19.349928 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337328 4603 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:19.349928 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337330 4603 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:19.349928 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337333 4603 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:19.349928 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337335 4603 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:19.349928 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337338 4603 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:19.349928 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337340 4603 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:19.349928 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337343 4603 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:19.349928 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.337368 4603 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:52:19.349928 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337401 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:19.349928 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337405 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:19.350424 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337408 4603 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:19.350424 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337412 4603 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:19.350424 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337414 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:19.350424 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337417 4603 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:19.350424 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337419 4603 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:19.350424 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337422 4603 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:19.350424 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337424 4603 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:19.350424 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337428 4603 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:19.350424 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337431 4603 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:19.350424 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337434 4603 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:19.350424 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337437 4603 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:19.350424 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337439 4603 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:19.350424 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337442 4603 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:19.350424 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337445 4603 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:19.350424 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337447 4603 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:19.350424 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337450 4603 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:19.350424 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337452 4603 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:19.350424 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337455 4603 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:19.350424 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337458 4603 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:19.350424 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337460 4603 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:19.350932 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337463 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:19.350932 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337466 4603 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:19.350932 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337469 4603 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:19.350932 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337472 4603 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:19.350932 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337474 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:19.350932 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337477 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:19.350932 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337479 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:19.350932 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337482 4603 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:19.350932 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337487 4603 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:19.350932 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337491 4603 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:19.350932 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337493 4603 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:19.350932 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337496 4603 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:19.350932 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337511 4603 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:19.350932 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337514 4603 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:19.350932 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337516 4603 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:19.350932 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337519 4603 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:19.350932 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337521 4603 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:19.350932 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337524 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:19.350932 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337527 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:19.351395 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337531 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:19.351395 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337533 4603 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:19.351395 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337536 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:19.351395 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337539 4603 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:19.351395 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337541 4603 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:19.351395 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337544 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:19.351395 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337546 4603 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:19.351395 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337549 4603 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:19.351395 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337552 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:19.351395 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337554 4603 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:19.351395 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337557 4603 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:19.351395 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337559 4603 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:19.351395 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337562 4603 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:19.351395 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337564 4603 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:19.351395 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337567 4603 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:19.351395 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337570 4603 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:19.351395 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337573 4603 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:19.351395 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337576 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:19.351395 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337579 4603 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:19.351997 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337582 4603 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:19.351997 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337585 4603 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:19.351997 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337589 4603 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:19.351997 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337592 4603 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:19.351997 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337594 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:19.351997 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337597 4603 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:19.351997 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337600 4603 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:19.351997 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337602 4603 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:19.351997 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337605 4603 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:19.351997 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337607 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:19.351997 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337610 4603 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:19.351997 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337613 4603 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:19.351997 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337615 4603 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:19.351997 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337619 4603 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:19.351997 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337622 4603 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:19.351997 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337625 4603 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:19.351997 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337627 4603 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:19.351997 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337630 4603 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:19.351997 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337632 4603 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:19.352456 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337636 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:19.352456 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337639 4603 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:19.352456 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337641 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:19.352456 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337644 4603 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:19.352456 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337646 4603 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:19.352456 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337649 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:19.352456 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337652 4603 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:19.352456 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337654 4603 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:19.352456 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337657 4603 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:19.352456 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337659 4603 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:19.352456 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337662 4603 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:19.352456 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337671 4603 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:19.352456 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337674 4603 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:19.352456 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337676 4603 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:19.352456 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337679 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:19.352456 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337681 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:19.352456 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337684 4603 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:19.352456 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337686 4603 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:19.352456 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337689 4603 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:19.352456 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337691 4603 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:19.352959 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337694 4603 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:19.352959 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337697 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:19.352959 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337699 4603 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:19.352959 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337702 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:19.352959 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337704 4603 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:19.352959 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337707 4603 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:19.352959 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.337712 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:19.352959 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.341375 4603 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 17:52:19.352959 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.341390 4603 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 17:52:19.352959 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.341408 4603 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 17:52:19.352959 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.341412 4603 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 17:52:19.352959 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341465 4603 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:19.352959 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341472 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:19.352959 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341475 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:19.352959 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341479 4603 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:19.352959 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341482 4603 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:19.352959 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341484 4603 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:19.352959 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341487 4603 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:19.352959 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341490 4603 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:19.353435 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341493 4603 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:19.353435 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341495 4603 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:19.353435 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341515 4603 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:19.353435 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341518 4603 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:19.353435 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341521 4603 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:19.353435 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341524 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:19.353435 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341527 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:19.353435 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341529 4603 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:19.353435 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341532 4603 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:19.353435 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341535 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:19.353435 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341538 4603 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:19.353435 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341540 4603 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:19.353435 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341543 4603 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:19.353435 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341546 4603 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:19.353435 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341548 4603 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:19.353435 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341551 4603 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:19.353435 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341553 4603 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:19.353435 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341556 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:19.353435 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341559 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:19.353435 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341562 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:19.353931 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341564 4603 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:19.353931 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341568 4603 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:19.353931 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341571 4603 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:19.353931 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341574 4603 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:19.353931 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341576 4603 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:19.353931 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341579 4603 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:19.353931 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341581 4603 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:19.353931 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341584 4603 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:19.353931 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341587 4603 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:19.353931 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341589 4603 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:19.353931 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341592 4603 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:19.353931 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341595 4603 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:19.353931 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341597 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:19.353931 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341600 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:19.353931 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341603 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:19.353931 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341605 4603 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:19.353931 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341608 4603 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:19.353931 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341610 4603 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:19.353931 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341613 4603 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:19.353931 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341615 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:19.354411 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341618 4603 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:19.354411 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341621 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:19.354411 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341624 4603 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:19.354411 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341627 4603 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:19.354411 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341629 4603 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:19.354411 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341632 4603 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:19.354411 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341635 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:19.354411 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341637 4603 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:19.354411 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341640 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:19.354411 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341643 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:19.354411 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341646 4603 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:19.354411 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341648 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:19.354411 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341651 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:19.354411 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341653 4603 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:19.354411 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341657 4603 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:19.354411 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341659 4603 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:19.354411 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341662 4603 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:19.354411 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341665 4603 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:19.354411 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341667 4603 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:19.354887 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341670 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:19.354887 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341672 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:19.354887 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341676 4603 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:19.354887 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341680 4603 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:19.354887 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341683 4603 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:19.354887 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341686 4603 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:19.354887 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341688 4603 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:19.354887 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341691 4603 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:19.354887 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341694 4603 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:19.354887 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341697 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:19.354887 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341699 4603 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:19.354887 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341702 4603 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:19.354887 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341706 4603 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:19.354887 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341708 4603 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:19.354887 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341711 4603 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:19.354887 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341714 4603 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:19.354887 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341716 4603 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:19.354887 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341719 4603 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:19.354887 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341722 4603 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:19.355345 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341724 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:19.355345 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341727 4603 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:19.355345 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341729 4603 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:19.355345 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341732 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:19.355345 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341735 4603 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:19.355345 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341737 4603 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:19.355345 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341740 4603 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:19.355345 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341742 4603 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:19.355345 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341745 4603 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:19.355345 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341748 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:19.355345 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341750 4603 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:19.355345 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341753 4603 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:19.355345 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341756 4603 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:19.355345 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341758 4603 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:19.355345 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341761 4603 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:19.355345 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341764 4603 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:19.355345 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341766 4603 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:19.355345 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341769 4603 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:19.355345 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341771 4603 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:19.355345 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341774 4603 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:19.355851 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.341801 4603 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:52:19.355851 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341943 4603 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:19.355851 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341948 4603 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:19.355851 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341952 4603 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:19.355851 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341955 4603 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:19.355851 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341958 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:19.355851 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341960 4603 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:19.355851 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341963 4603 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:19.355851 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341965 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:19.355851 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341968 4603 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:19.355851 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341970 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:19.355851 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341973 4603 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:19.355851 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341975 4603 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:19.355851 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341978 4603 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:19.355851 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341981 4603 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:19.355851 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341983 4603 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:19.356230 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341986 4603 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:19.356230 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341988 4603 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:19.356230 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341991 4603 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:19.356230 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341993 4603 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:19.356230 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.341996 4603 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:19.356230 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342005 4603 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:19.356230 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342010 4603 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:19.356230 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342012 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:19.356230 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342015 4603 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:19.356230 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342018 4603 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:19.356230 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342020 4603 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:19.356230 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342023 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:19.356230 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342025 4603 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:19.356230 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342028 4603 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:19.356230 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342030 4603 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:19.356230 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342033 4603 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:19.356230 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342036 4603 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:19.356230 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342038 4603 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:19.356230 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342041 4603 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:19.356230 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342043 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:19.356706 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342046 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:19.356706 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342049 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:19.356706 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342051 4603 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:19.356706 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342054 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:19.356706 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342057 4603 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:19.356706 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342059 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:19.356706 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342062 4603 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:19.356706 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342064 4603 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:19.356706 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342067 4603 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:19.356706 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342069 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:19.356706 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342072 4603 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:19.356706 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342074 4603 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:19.356706 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342077 4603 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:19.356706 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342079 4603 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:19.356706 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342082 4603 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:19.356706 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342085 4603 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:19.356706 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342088 4603 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:19.356706 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342090 4603 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:19.356706 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342093 4603 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:19.357152 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342096 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:19.357152 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342098 4603 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:19.357152 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342101 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:19.357152 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342103 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:19.357152 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342106 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:19.357152 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342108 4603 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:19.357152 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342111 4603 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:19.357152 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342113 4603 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:19.357152 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342115 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:19.357152 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342118 4603 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:19.357152 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342120 4603 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:19.357152 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342123 4603 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:19.357152 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342126 4603 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:19.357152 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342128 4603 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:19.357152 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342131 4603 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:19.357152 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342133 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:19.357152 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342135 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:19.357152 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342138 4603 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:19.357152 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342140 4603 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:19.357620 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342143 4603 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:19.357620 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342145 4603 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:19.357620 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342148 4603 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:19.357620 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342150 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:19.357620 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342153 4603 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:19.357620 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342155 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:19.357620 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342158 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:19.357620 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342160 4603 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:19.357620 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342163 4603 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:19.357620 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342165 4603 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:19.357620 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342168 4603 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:19.357620 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342170 4603 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:19.357620 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342173 4603 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:19.357620 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342175 4603 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:19.357620 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342178 4603 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:19.357620 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342181 4603 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:19.357620 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342185 4603 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:19.357620 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342189 4603 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:19.357620 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342193 4603 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:19.358072 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342196 4603 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:19.358072 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342200 4603 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:19.358072 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342202 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:19.358072 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342205 4603 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:19.358072 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342208 4603 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:19.358072 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342211 4603 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:19.358072 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342213 4603 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:19.358072 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342216 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:19.358072 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342219 4603 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:19.358072 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342221 4603 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:19.358072 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342224 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:19.358072 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342226 4603 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:19.358072 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342229 4603 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:19.358072 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342231 4603 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:19.358072 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.342253 4603 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:52:19.358072 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342283 4603 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:19.358441 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342287 4603 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:19.358441 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342290 4603 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:19.358441 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342292 4603 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:19.358441 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342295 4603 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:19.358441 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342298 4603 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:19.358441 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342300 4603 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:19.358441 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342303 4603 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:19.358441 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342305 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:19.358441 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342308 4603 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:19.358441 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342310 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:19.358441 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342313 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:19.358441 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342315 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:19.358441 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342318 4603 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:19.358441 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342321 4603 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:19.358441 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342324 4603 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:19.358441 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342326 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:19.358441 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342328 4603 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:19.358441 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342331 4603 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:19.358441 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342334 4603 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:19.358939 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342336 4603 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:19.358939 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342338 4603 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:19.358939 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342341 4603 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:19.358939 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342348 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:19.358939 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342351 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:19.358939 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342354 4603 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:19.358939 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342356 4603 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:19.358939 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342359 4603 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:19.358939 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342361 4603 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:19.358939 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342364 4603 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:19.358939 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342367 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:19.358939 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342369 4603 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:19.358939 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342372 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:19.358939 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342375 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:19.358939 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342377 4603 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:19.358939 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342380 4603 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:19.358939 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342383 4603 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:19.358939 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342385 4603 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:19.358939 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342388 4603 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:19.359393 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342391 4603 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:19.359393 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342394 4603 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:19.359393 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342397 4603 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:19.359393 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342399 4603 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:19.359393 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342403 4603 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:19.359393 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342407 4603 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:19.359393 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342411 4603 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:19.359393 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342416 4603 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:19.359393 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342419 4603 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:19.359393 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342422 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:19.359393 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342425 4603 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:19.359393 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342427 4603 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:19.359393 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342430 4603 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:19.359393 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342433 4603 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:19.359393 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342435 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:19.359393 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342438 4603 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:19.359393 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342442 4603 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:19.359393 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342444 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:19.359393 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342447 4603 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:19.359968 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342450 4603 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:19.359968 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342452 4603 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:19.359968 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342455 4603 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:19.359968 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342457 4603 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:19.359968 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342460 4603 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:19.359968 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342462 4603 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:19.359968 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342465 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:19.359968 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342468 4603 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:19.359968 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342470 4603 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:19.359968 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342473 4603 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:19.359968 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342475 4603 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:19.359968 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342478 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:19.359968 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342480 4603 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:19.359968 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342483 4603 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:19.359968 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342485 4603 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:19.359968 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342488 4603 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:19.359968 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342490 4603 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:19.359968 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342493 4603 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:19.359968 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342495 4603 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:19.359968 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342510 4603 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:19.360583 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342513 4603 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:19.360583 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342516 4603 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:19.360583 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342520 4603 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:19.360583 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342522 4603 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:19.360583 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342525 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:19.360583 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342527 4603 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:19.360583 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342530 4603 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:19.360583 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342532 4603 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:19.360583 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342535 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:19.360583 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342539 4603 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:19.360583 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342542 4603 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:19.360583 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342544 4603 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:19.360583 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342547 4603 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:19.360583 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342550 4603 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:19.360583 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342552 4603 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:19.360583 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342555 4603 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:19.360583 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342557 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:19.360583 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342560 4603 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:19.360583 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342562 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:19.360583 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342565 4603 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:19.361237 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342568 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:19.361237 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342570 4603 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:19.361237 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342573 4603 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:19.361237 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342576 4603 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:19.361237 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342578 4603 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:19.361237 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342581 4603 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:19.361237 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342583 4603 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:19.361237 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:19.342587 4603 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:19.361237 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.342641 4603 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 17:52:19.361237 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.343555 4603 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 17:52:19.361237 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.343599 4603 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 17:52:19.361237 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.343784 4603 server.go:1014] "Starting client certificate rotation" Oct 30 17:52:19.361237 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.343944 4603 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:52:19.361237 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.344085 4603 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:46:41 +0000 UTC" deadline="2028-06-15 16:33:33.273990735 +0000 UTC" Oct 30 17:52:19.361237 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.344129 4603 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14254h41m13.929868632s" Oct 30 17:52:19.361237 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.344766 4603 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:52:19.361902 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.346422 4603 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:52:19.361902 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.347722 4603 log.go:25] "Validated CRI v1 runtime API" Oct 30 17:52:19.361902 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.349204 4603 log.go:25] "Validated CRI v1 image API" Oct 30 17:52:19.361902 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.349952 4603 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 17:52:19.361902 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.351456 4603 fs.go:135] Filesystem UUIDs: map[0ebb35b0-35ed-47cf-8a34-7459d7fb119c:/dev/nvme0n1p3 25cf32f3-6603-46f4-9061-490321d7fddf:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 30 17:52:19.362146 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.351475 4603 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/038cb98c16682576ea3b21b367e72ebda6dfeafaf0e4643e705aff56bff3bebc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/038cb98c16682576ea3b21b367e72ebda6dfeafaf0e4643e705aff56bff3bebc/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b62de9d5d2c99cd9c58cb7336c9808356ca0c869635fbc2db41483bdeeea170/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b62de9d5d2c99cd9c58cb7336c9808356ca0c869635fbc2db41483bdeeea170/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f19baba013a313e698e143c78145a5d6e27e42801250497ffdf8688c0de5980/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f19baba013a313e698e143c78145a5d6e27e42801250497ffdf8688c0de5980/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/44ab6da95edbd8941f700dce2e52925fdd282921db1dab6c2230bc9d3203dbf4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/44ab6da95edbd8941f700dce2e52925fdd282921db1dab6c2230bc9d3203dbf4/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/469f74b30406a1a093d82860170c4647c39ad02766381d22db7a00389fa9ac0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/469f74b30406a1a093d82860170c4647c39ad02766381d22db7a00389fa9ac0b/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/576a48cc07912595cac16d588f2857199ee4b0c8bc0b940d33575fe3873609c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/576a48cc07912595cac16d588f2857199ee4b0c8bc0b940d33575fe3873609c7/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81c432afc2475543bfeda559a25113dc69a47425c99a1a739b7416bf9acf983d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81c432afc2475543bfeda559a25113dc69a47425c99a1a739b7416bf9acf983d/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88566190f852aad7ef96b6cdae8fa8b8c293ceb99d24ae02cfa03e50341a9400/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88566190f852aad7ef96b6cdae8fa8b8c293ceb99d24ae02cfa03e50341a9400/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba0da1b91d88008228eaa5ff170fdec70cf0497ac615c8e1ec2f021f626f49f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba0da1b91d88008228eaa5ff170fdec70cf0497ac615c8e1ec2f021f626f49f1/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd5a6a5bef5bd27b9d1982aabd2157ac61ec13df9888f62fb9027c666fcfbe0e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd5a6a5bef5bd27b9d1982aabd2157ac61ec13df9888f62fb9027c666fcfbe0e/userdata/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bfcd9bd03fb23279901627f01b6de1508da2d512c4161a5062b50f5dd429d5ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bfcd9bd03fb23279901627f01b6de1508da2d512c4161a5062b50f5dd429d5ae/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c3f671110e035bc547a056600fd7cc028da094c7c56c804695f0eeacab6992e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c3f671110e035bc547a056600fd7cc028da094c7c56c804695f0eeacab6992e2/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf000717c4c846facb91c7480cc7c5102ec411c4615ce12d44b92cccb8724127/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf000717c4c846facb91c7480cc7c5102ec411c4615ce12d44b92cccb8724127/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0aa89f80635d0c18e95244519e81178a0752ad297c369bb202fde20a4473346/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0aa89f80635d0c18e95244519e81178a0752ad297c369bb202fde20a4473346/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da9b5034a6309885bd5ae6b90fbe1356749fb6c830805e9c3387fafacf3aa71a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da9b5034a6309885bd5ae6b90fbe1356749fb6c830805e9c3387fafacf3aa71a/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3a38c5522ed62d2130d747779834a8925720b6d48468a3d3e0271fc3d656874/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3a38c5522ed62d2130d747779834a8925720b6d48468a3d3e0271fc3d656874/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2ac51125accf1e753569ce7a42fd3450827e17dc4c78b239189f6d78ae5973c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2ac51125accf1e753569ce7a42fd3450827e17dc4c78b239189f6d78ae5973c/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0/volumes/kubernetes.io~projected/kube-api-access-wzf62:{mountpoint:/var/lib/kubelet/pods/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0/volumes/kubernetes.io~projected/kube-api-access-wzf62 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/088d4d32-c02b-4da2-9435-c5bc22352079/volumes/kubernetes.io~projected/kube-api-access-4t2rp:{mountpoint:/var/lib/kubelet/pods/088d4d32-c02b-4da2-9435-c5bc22352079/volumes/kubernetes.io~projected/kube-api-access-4t2rp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a6fe435-1778-45b0-ae18-6695f5824229/volumes/kubernetes.io~projected/kube-api-access-z85fd:{mountpoint:/var/lib/kubelet/pods/1a6fe435-1778-45b0-ae18-6695f5824229/volumes/kubernetes.io~projected/kube-api-access-z85fd major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e6590e8-bfc4-4b19-b579-7180aa6af22a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/1e6590e8-bfc4-4b19-b579-7180aa6af22a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21a54420-0e75-4d4b-93cd-fe72c0cc2070/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/21a54420-0e75-4d4b-93cd-fe72c0cc2070/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21a54420-0e75-4d4b-93cd-fe72c0cc2070/volumes/kubernetes.io~projected/kube-api-access-hmwhl:{mountpoint:/var/lib/kubelet/pods/21a54420-0e75-4d4b-93cd-fe72c0cc2070/volumes/kubernetes.io~projected/kube-api-access-hmwhl major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21a54420-0e75-4d4b-93cd-fe72c0cc2070/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/21a54420-0e75-4d4b-93cd-fe72c0cc2070/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21a54420-0e75-4d4b-93cd-fe72c0cc2070/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/21a54420-0e75-4d4b-93cd-fe72c0cc2070/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/355e77f7-f944-4955-95c2-bdf4c7ea12b7/volumes/kubernetes.io~projected/kube-api-access-d4bqp:{mountpoint:/var/lib/kubelet/pods/355e77f7-f944-4955-95c2-bdf4c7ea12b7/volumes/kubernetes.io~projected/kube-api-access-d4bqp major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ef04039-ef85-40ef-8f30-a0edd7b402e0/volumes/kubernetes.io~projected/kube-api-access-lm6vb:{mountpoint:/var/lib/kubelet/pods/3ef04039-ef85-40ef-8f30-a0edd7b402e0/volumes/kubernetes.io~projected/kube-api-access-lm6vb major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/471ac5c3-1cdd-4d79-bdc1-3acce27cc042/volumes/kubernetes.io~projected/kube-api-access-8ks5l:{mountpoint:/var/lib/kubelet/pods/471ac5c3-1cdd-4d79-bdc1-3acce27cc042/volumes/kubernetes.io~projected/kube-api-access-8ks5l major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72993a50-bf5d-40c5-833b-afff9f86e61d/volumes/kubernetes.io~projected/kube-api-access-lxtq4:{mountpoint:/var/lib/kubelet/pods/72993a50-bf5d-40c5-833b-afff9f86e61d/volumes/kubernetes.io~projected/kube-api-access-lxtq4 major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7af86f87-5d83-4a8b-a0cc-88d4641406d2/volumes/kubernetes.io~projected/kube-api-access-vq24z:{mountpoint:/var/lib/kubelet/pods/7af86f87-5d83-4a8b-a0cc-88d4641406d2/volumes/kubernetes.io~projected/kube-api-access-vq24z major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/827bbba0-b306-43db-b805-8c87dfabb874/volumes/kubernetes.io~projected/kube-api-access-nj9k2:{mountpoint:/var/lib/kubelet/pods/827bbba0-b306-43db-b805-8c87dfabb874/volumes/kubernetes.io~projected/kube-api-access-nj9k2 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e818352-df23-46b4-84d9-81fd259200ec/volumes/kubernetes.io~projected/kube-api-access-4qplp:{mountpoint:/var/lib/kubelet/pods/8e818352-df23-46b4-84d9-81fd259200ec/volumes/kubernetes.io~projected/kube-api-access-4qplp major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90437f91-1657-47db-b3a1-d59bc8d1de14/volumes/kubernetes.io~projected/kube-api-access-fbs6z:{mountpoint:/var/lib/kubelet/pods/90437f91-1657-47db-b3a1-d59bc8d1de14/volumes/kubernetes.io~projected/kube-api-access-fbs6z major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95709b19-3c85-4ef1-9110-c23a97847732/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/95709b19-3c85-4ef1-9110-c23a97847732/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95709b19-3c85-4ef1-9110-c23a97847732/volumes/kubernetes.io~projected/kube-api-access-dhlzv:{mountpoint:/var/lib/kubelet/pods/95709b19-3c85-4ef1-9110-c23a97847732/volumes/kubernetes.io~projected/kube-api-access-dhlzv major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95709b19-3c85-4ef1-9110-c23a97847732/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/95709b19-3c85-4ef1-9110-c23a97847732/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63/volumes/kubernetes.io~projected/kube-api-access-vgxhb:{mountpoint:/var/lib/kubelet/pods/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63/volumes/kubernetes.io~projected/kube-api-access-vgxhb major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e/volumes/kubernetes.io~projected/kube-api-access-qbc66:{mountpoint:/var/lib/kubelet/pods/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e/volumes/kubernetes.io~projected/kube-api-access-qbc66 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9602b28-165a-4c17-96fa-9f0e25fd2cf8/volumes/kubernetes.io~projected/kube-api-access-r9xxt:{mountpoint:/var/lib/kubelet/pods/c9602b28-165a-4c17-96fa-9f0e25fd2cf8/volumes/kubernetes.io~projected/kube-api-access-r9xxt major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9602b28-165a-4c17-96fa-9f0e25fd2cf8/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/c9602b28-165a-4c17-96fa-9f0e25fd2cf8/volumes/kubernetes.io~secret/cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d58a06a1-fdbc-439e-8572-fdeabc6ee4ef/volumes/kubernetes.io~projected/kube-api-access-ll92d:{mountpoint:/var/lib/kubelet/pods/d58a06a1-fdbc-439e-8572-fdeabc6ee4ef/volumes/kubernetes.io~projected/kube-api-access-ll92d major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9933e90-d394-4e89-8e0f-115298c359fa/volumes/kubernetes.io~projected/kube-api-access-2lnnm:{mountpoint:/var/lib/kubelet/pods/e9933e90-d394-4e89-8e0f-115298c359fa/volumes/kubernetes.io~projected/kube-api-access-2lnnm major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd6cf358-80f3-42f6-9848-6386a937a57a/volumes/kubernetes.io~projected/kube-api-access-pk78z:{mountpoint:/var/lib/kubelet/pods/fd6cf358-80f3-42f6-9848-6386a937a57a/volumes/kubernetes.io~projected/kube-api-access-pk78z major:0 minor:64 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/b78d9b75625259bbdd17f4b800b45ed68b50b76faf5d882365fac819e8dafa84/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e8892190f583daed6fd1be291b2fb1f85505fd198eb6d58bc553d7b3a1c1ad0d/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/c54af24cd163abb31f9ca8c3eac7ebe41da144a4b2806c516b97d72904702826/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/fab782109be6f569431ea6fa4012d86cc9d615fe70bf556d4a9c0e5e5d6225a7/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/f722bfa501d73230f4c167ec8de1887c12997461420f19239d4822dd4d588f2f/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/7dacaa9a895a001652ae336d8389c377ec57d071fdb5785e3a8d40e0392f9b8b/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/82f9768346ffb2dabcf5dac882e8dc5829d0d17e95c859752e4ac510755313e8/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e4980b57fae46b8440627826e07e3c0f2c691c870b2c3b697020345f2f77d3c5/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/c45b9b552c5bc1fe01358523c9d4eb994cd0939405eff8bfc59f8681d1cb9e75/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/b28a04187a1fd0906ddf8878ffc546948131325ccb6d07d71a1b45d07587005f/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/2336b45ed380bcce5580f78b2a40331f19e3a738daae0dd786e78a66b57e5f90/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/df17d6cac9b91ceaafbe63c7eb1f20111c9b809f872033e47df5f5f413e764cc/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/f1c2694f0c4f45c8bdc60ada5048f879635cae7d6f04ea0288e2bcefd4f52fe2/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/7982ae9f9017093c2090b19b35cb33a100676ddf3c6c0829539c1af845303700/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/f343bcbc0852499c343fde076e6038f09b6e09eba57bf13c8a27c4f7fed6db09/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/f7779a10f7893a1e587afd304081728cfc6ebcaf0259c5a059de6383ae1e4ebb/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/1888e2f3e6c628a3dbbbad289ce9233e5ed1623299e39b876d7e406627d40f2c/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/da2ae0f5dc51dca6abccc84f64b47b52a746194a6d26c92986ad8f8cd83fc808/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/8888f27c1d0a61b344bcdfa4d78c89c1e19a5fa353f7137730a953ea5d4d2ad9/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/006f456261c702766716a3e3c7aff4802b49c2a33c51f41a4a3422bddd36992f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/1e7b7d1533f9a38b75b8576dac86afd6baec7f87f03c8b8a932170e70b7bf873/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/7baeed2a9ed603f00b5ceb65aec49c19cb2bf32700911be885dbec9727bd4cbe/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/48a3e05ddf9325b1e4326ccbba163ada2d7e8d25f622ed40a45d4fb83b538eda/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/81958f040d418abf42536cea5986c3c58da5c8bbc48f00364cd49a4b0d6d7f5f/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/d35fd9397c8c658a661c224843d34366313ac12ff0417a7b3b7147ad6bc68e12/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/8443b3322cbf55cec994df933a2f26b9d0b84cb8274d16a7d76015e17563818d/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/8b969bd49c42f49f87fc4af477eb8a37a1324bd57672ea9a2b28733f7feb57f2/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/bced96c3cd6f23a9ae53c160e6cc98470f24ff96aca8da34ced17ca65e328b86/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/42e52c12b784fe2dc1944fe52b0ddf95d67be0112db019f9ad443510ea652ac6/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/c01cd9c7596ddb8f30220d8ba2655c7f2289401c904aa2bf5be7288c479ad382/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/92032f4333ee7053ea6d3775159315539ce9d3d42aeefa1a490aaf9dd5c83fd5/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/5f4a3abafe491fe244935cab0316d7c5868de92f2de4302623645d4f44ca3380/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/da5d0868975f54096c75037498db0a8f9b00c799677c4641fd2d9d67bf5e3752/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/2aa09befa4d6a4b80158590e4f20ba24158d81e46595cd8a6ee8d0b588abe508/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/4a1818cb61064f1caa7eb29c0e8089cae41e80d4d2197d77581e13bf58790602/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-50:{mountpoint:/var/lib/containers/storage/overlay/865bd4c88dd1e8cdaea7a8f3163a94de8a3eab607b80fe70180ba2f263e16720/merged major:0 minor:50 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/3e45d0bfef715c55e8fe5fe50d08b327d0b87dd5dc43b473d7905a37897767b6/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/370a75f1baf3e010ec3d62c819d276f2097afce1dc1f8581fee4cf59603416b4/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/ef064e33155a950f6420955e52e2e0b3cfe8ec64bdf1a8a579057bf340750186/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/46fe02eb82eec64196afe0633fdb348a8763513ed2fd8f55b982f31c603c306a/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/c041c822ee5672995ff713ba13301de8bbf776872d158a0d976f27ace59afe22/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/9b12b072a8c3b6e8667d60321f1d5e93e948b7ee8a319f19dc5204508e40ce14/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/66141a40fd8f3762dc3b39c41f6a1747136b361132dd8c3c2264cc79e3f35cb3/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/ae9a64c7ca6d6d7202f1b0a9c8aeb1eacf7fad55ff6653e2b255ccb10c3b4156/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/c25ad70739be6b44f664f76e26bea1a2c1724b28780e7647f6fa1bb6ea4be74d/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 17:52:19.362417 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.358339 4603 manager.go:217] Machine: {Timestamp:2026-10-30 17:52:19.357888451 +0000 UTC m=+0.073458316 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099902 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ce0bb041ebbb460e8fc4e4f86656a SystemUUID:ec2ce0bb-041e-bbb4-60e8-fc4e4f86656a BootID:53cf46c1-e095-45b3-860f-cf500870b47a Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da9b5034a6309885bd5ae6b90fbe1356749fb6c830805e9c3387fafacf3aa71a/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9602b28-165a-4c17-96fa-9f0e25fd2cf8/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/21a54420-0e75-4d4b-93cd-fe72c0cc2070/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88566190f852aad7ef96b6cdae8fa8b8c293ceb99d24ae02cfa03e50341a9400/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba0da1b91d88008228eaa5ff170fdec70cf0497ac615c8e1ec2f021f626f49f1/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e6590e8-bfc4-4b19-b579-7180aa6af22a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:232 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/355e77f7-f944-4955-95c2-bdf4c7ea12b7/volumes/kubernetes.io~projected/kube-api-access-d4bqp DeviceMajor:0 DeviceMinor:210 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90437f91-1657-47db-b3a1-d59bc8d1de14/volumes/kubernetes.io~projected/kube-api-access-fbs6z DeviceMajor:0 DeviceMinor:237 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2ac51125accf1e753569ce7a42fd3450827e17dc4c78b239189f6d78ae5973c/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bfcd9bd03fb23279901627f01b6de1508da2d512c4161a5062b50f5dd429d5ae/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72993a50-bf5d-40c5-833b-afff9f86e61d/volumes/kubernetes.io~projected/kube-api-access-lxtq4 DeviceMajor:0 DeviceMinor:273 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd5a6a5bef5bd27b9d1982aabd2157ac61ec13df9888f62fb9027c666fcfbe0e/userdata/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3ef04039-ef85-40ef-8f30-a0edd7b402e0/volumes/kubernetes.io~projected/kube-api-access-lm6vb DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/471ac5c3-1cdd-4d79-bdc1-3acce27cc042/volumes/kubernetes.io~projected/kube-api-access-8ks5l DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/088d4d32-c02b-4da2-9435-c5bc22352079/volumes/kubernetes.io~projected/kube-api-access-4t2rp DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c3f671110e035bc547a056600fd7cc028da094c7c56c804695f0eeacab6992e2/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3a38c5522ed62d2130d747779834a8925720b6d48468a3d3e0271fc3d656874/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e/volumes/kubernetes.io~projected/kube-api-access-qbc66 DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21a54420-0e75-4d4b-93cd-fe72c0cc2070/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/469f74b30406a1a093d82860170c4647c39ad02766381d22db7a00389fa9ac0b/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0/volumes/kubernetes.io~projected/kube-api-access-wzf62 DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b62de9d5d2c99cd9c58cb7336c9808356ca0c869635fbc2db41483bdeeea170/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e818352-df23-46b4-84d9-81fd259200ec/volumes/kubernetes.io~projected/kube-api-access-4qplp DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7af86f87-5d83-4a8b-a0cc-88d4641406d2/volumes/kubernetes.io~projected/kube-api-access-vq24z DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95709b19-3c85-4ef1-9110-c23a97847732/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-50 DeviceMajor:0 DeviceMinor:50 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/827bbba0-b306-43db-b805-8c87dfabb874/volumes/kubernetes.io~projected/kube-api-access-nj9k2 DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e9933e90-d394-4e89-8e0f-115298c359fa/volumes/kubernetes.io~projected/kube-api-access-2lnnm DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fd6cf358-80f3-42f6-9848-6386a937a57a/volumes/kubernetes.io~projected/kube-api-access-pk78z DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21a54420-0e75-4d4b-93cd-fe72c0cc2070/volumes/kubernetes.io~projected/kube-api-access-hmwhl DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63/volumes/kubernetes.io~projected/kube-api-access-vgxhb DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21a54420-0e75-4d4b-93cd-fe72c0cc2070/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c9602b28-165a-4c17-96fa-9f0e25fd2cf8/volumes/kubernetes.io~projected/kube-api-access-r9xxt DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/44ab6da95edbd8941f700dce2e52925fdd282921db1dab6c2230bc9d3203dbf4/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a6fe435-1778-45b0-ae18-6695f5824229/volumes/kubernetes.io~projected/kube-api-access-z85fd DeviceMajor:0 DeviceMinor:266 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/95709b19-3c85-4ef1-9110-c23a97847732/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/576a48cc07912595cac16d588f2857199ee4b0c8bc0b940d33575fe3873609c7/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf000717c4c846facb91c7480cc7c5102ec411c4615ce12d44b92cccb8724127/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0aa89f80635d0c18e95244519e81178a0752ad297c369bb202fde20a4473346/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f19baba013a313e698e143c78145a5d6e27e42801250497ffdf8688c0de5980/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d58a06a1-fdbc-439e-8572-fdeabc6ee4ef/volumes/kubernetes.io~projected/kube-api-access-ll92d DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/038cb98c16682576ea3b21b367e72ebda6dfeafaf0e4643e705aff56bff3bebc/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/81c432afc2475543bfeda559a25113dc69a47425c99a1a739b7416bf9acf983d/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/95709b19-3c85-4ef1-9110-c23a97847732/volumes/kubernetes.io~projected/kube-api-access-dhlzv DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:038cb98c1668257 MacAddress:d6:b6:da:90:b4:9b Speed:10000 Mtu:8901} {Name:44ab6da95edbd89 MacAddress:16:11:80:aa:0a:b3 Speed:10000 Mtu:8901} {Name:469f74b30406a1a MacAddress:4e:d0:e6:b1:c9:9c Speed:10000 Mtu:8901} {Name:576a48cc0791259 MacAddress:8a:29:18:44:6d:66 Speed:10000 Mtu:8901} {Name:88566190f852aad MacAddress:0a:9e:7f:6f:c8:23 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c0:49:dc:bf Speed:0 Mtu:9001} {Name:br-int MacAddress:12:40:7f:77:0d:9a Speed:0 Mtu:8901} {Name:cf000717c4c846f MacAddress:be:97:fb:fa:4d:b1 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c0:49:dc:bf Speed:0 Mtu:9001} {Name:f2ac51125accf1e MacAddress:22:73:bd:3f:06:86 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:36:37:dd:74:9a:4f Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:2e:bf:2c:7f:db:05 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 17:52:19.362417 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.358621 4603 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 17:52:19.362417 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.358659 4603 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 17:52:19.362417 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.358912 4603 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 17:52:19.362417 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.358934 4603 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-236.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 17:52:19.362417 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.359045 4603 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 17:52:19.362417 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.359052 4603 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 17:52:19.362417 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.359066 4603 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:52:19.362417 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.359079 4603 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:52:19.362417 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.359146 4603 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 17:52:19.362417 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.359298 4603 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 17:52:19.362417 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.359391 4603 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 17:52:19.362417 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.359423 4603 kubelet.go:493] "Attempting to sync node with API server" Oct 30 17:52:19.362417 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.359433 4603 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 17:52:19.362417 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.359441 4603 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 17:52:19.362417 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.359450 4603 kubelet.go:405] "Adding apiserver pod source" Oct 30 17:52:19.362417 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.359458 4603 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 17:52:19.362417 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.360005 4603 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 17:52:19.362417 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.360015 4603 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:52:19.362417 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.360446 4603 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 17:52:19.364699 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.364680 4603 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 17:52:19.364745 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.364701 4603 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:52:19.364829 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.364816 4603 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 17:52:19.365091 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.365077 4603 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 17:52:19.365132 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.365122 4603 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 17:52:19.365253 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.365242 4603 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 17:52:19.365282 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.365262 4603 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 17:52:19.365282 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.365273 4603 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 17:52:19.365282 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.365281 4603 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 17:52:19.365353 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.365291 4603 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 17:52:19.365353 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.365300 4603 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 17:52:19.365353 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.365309 4603 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 17:52:19.365353 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.365318 4603 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 17:52:19.365353 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.365328 4603 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 17:52:19.365353 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.365337 4603 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 17:52:19.365353 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.365349 4603 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 17:52:19.365546 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.365363 4603 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 17:52:19.365546 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.365398 4603 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 17:52:19.365546 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.365407 4603 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 17:52:19.368665 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.368652 4603 server.go:1263] "Started kubelet" Oct 30 17:52:19.368750 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.368698 4603 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 17:52:19.368823 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.368748 4603 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 17:52:19.368872 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.368849 4603 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 17:52:19.369964 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.369147 4603 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 17:52:19.369413 ip-10-0-128-236 systemd[1]: Started Kubernetes Kubelet. Oct 30 17:52:19.370164 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.370061 4603 server.go:317] "Adding debug handlers to kubelet server" Oct 30 17:52:19.372477 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.372236 4603 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 17:52:19.372477 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.372434 4603 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 17:52:19.375642 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.375600 4603 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:46:42 +0000 UTC" deadline="2028-07-25 16:09:07.792320484 +0000 UTC" Oct 30 17:52:19.376873 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.376855 4603 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15214h16m48.415471295s" Oct 30 17:52:19.377136 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.375918 4603 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 17:52:19.377228 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.377217 4603 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 17:52:19.377378 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.375936 4603 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 17:52:19.377470 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:19.375825 4603 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-236.ec2.internal\" not found" Oct 30 17:52:19.377560 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.376688 4603 factory.go:55] Registering systemd factory Oct 30 17:52:19.377635 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.377626 4603 factory.go:223] Registration of the systemd container factory successfully Oct 30 17:52:19.379001 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.378524 4603 factory.go:153] Registering CRI-O factory Oct 30 17:52:19.379001 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.378543 4603 factory.go:223] Registration of the crio container factory successfully Oct 30 17:52:19.379001 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.378605 4603 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 17:52:19.379001 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.378626 4603 factory.go:103] Registering Raw factory Oct 30 17:52:19.379001 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.378641 4603 manager.go:1199] Started watching for new ooms in manager Oct 30 17:52:19.379188 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.379037 4603 manager.go:319] Starting recovery of all containers Oct 30 17:52:19.384904 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.384733 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd6cf358-80f3-42f6-9848-6386a937a57a" volumeName="kubernetes.io/projected/fd6cf358-80f3-42f6-9848-6386a937a57a-kube-api-access-pk78z" seLinuxMountContext="" Oct 30 17:52:19.384904 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.384769 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="038ae7b2-90e2-44e6-9437-fcb4a0bcfda0" volumeName="kubernetes.io/empty-dir/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-tmp" seLinuxMountContext="" Oct 30 17:52:19.384904 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.384805 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21a54420-0e75-4d4b-93cd-fe72c0cc2070" volumeName="kubernetes.io/configmap/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-certificates" seLinuxMountContext="" Oct 30 17:52:19.384904 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.384820 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21a54420-0e75-4d4b-93cd-fe72c0cc2070" volumeName="kubernetes.io/empty-dir/21a54420-0e75-4d4b-93cd-fe72c0cc2070-ca-trust-extracted" seLinuxMountContext="" Oct 30 17:52:19.384904 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.384833 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21a54420-0e75-4d4b-93cd-fe72c0cc2070" volumeName="kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-kube-api-access-hmwhl" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.384851 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e818352-df23-46b4-84d9-81fd259200ec" volumeName="kubernetes.io/configmap/8e818352-df23-46b4-84d9-81fd259200ec-config-volume" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385152 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63" volumeName="kubernetes.io/configmap/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-trusted-ca" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385175 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9602b28-165a-4c17-96fa-9f0e25fd2cf8" volumeName="kubernetes.io/projected/c9602b28-165a-4c17-96fa-9f0e25fd2cf8-kube-api-access-r9xxt" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385204 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d58a06a1-fdbc-439e-8572-fdeabc6ee4ef" volumeName="kubernetes.io/projected/d58a06a1-fdbc-439e-8572-fdeabc6ee4ef-kube-api-access-ll92d" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385216 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21a54420-0e75-4d4b-93cd-fe72c0cc2070" volumeName="kubernetes.io/configmap/21a54420-0e75-4d4b-93cd-fe72c0cc2070-trusted-ca" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385228 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21a54420-0e75-4d4b-93cd-fe72c0cc2070" volumeName="kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-bound-sa-token" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385244 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21a54420-0e75-4d4b-93cd-fe72c0cc2070" volumeName="kubernetes.io/secret/21a54420-0e75-4d4b-93cd-fe72c0cc2070-image-registry-private-configuration" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385257 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ef04039-ef85-40ef-8f30-a0edd7b402e0" volumeName="kubernetes.io/empty-dir/3ef04039-ef85-40ef-8f30-a0edd7b402e0-tmp-dir" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385269 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ef04039-ef85-40ef-8f30-a0edd7b402e0" volumeName="kubernetes.io/projected/3ef04039-ef85-40ef-8f30-a0edd7b402e0-kube-api-access-lm6vb" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385285 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9933e90-d394-4e89-8e0f-115298c359fa" volumeName="kubernetes.io/configmap/e9933e90-d394-4e89-8e0f-115298c359fa-iptables-alerter-script" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385299 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd6cf358-80f3-42f6-9848-6386a937a57a" volumeName="kubernetes.io/configmap/fd6cf358-80f3-42f6-9848-6386a937a57a-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385312 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd6cf358-80f3-42f6-9848-6386a937a57a" volumeName="kubernetes.io/configmap/fd6cf358-80f3-42f6-9848-6386a937a57a-cni-binary-copy" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385323 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="038ae7b2-90e2-44e6-9437-fcb4a0bcfda0" volumeName="kubernetes.io/projected/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-kube-api-access-wzf62" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385336 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e6590e8-bfc4-4b19-b579-7180aa6af22a" volumeName="kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385350 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72993a50-bf5d-40c5-833b-afff9f86e61d" volumeName="kubernetes.io/projected/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-api-access-lxtq4" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385380 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90437f91-1657-47db-b3a1-d59bc8d1de14" volumeName="kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385397 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63" volumeName="kubernetes.io/configmap/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-certificates" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385410 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63" volumeName="kubernetes.io/empty-dir/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-ca-trust-extracted" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385422 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9933e90-d394-4e89-8e0f-115298c359fa" volumeName="kubernetes.io/projected/e9933e90-d394-4e89-8e0f-115298c359fa-kube-api-access-2lnnm" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385436 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a6fe435-1778-45b0-ae18-6695f5824229" volumeName="kubernetes.io/projected/1a6fe435-1778-45b0-ae18-6695f5824229-kube-api-access-z85fd" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385449 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="355e77f7-f944-4955-95c2-bdf4c7ea12b7" volumeName="kubernetes.io/projected/355e77f7-f944-4955-95c2-bdf4c7ea12b7-kube-api-access-d4bqp" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385463 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="471ac5c3-1cdd-4d79-bdc1-3acce27cc042" volumeName="kubernetes.io/projected/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-kube-api-access-8ks5l" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385476 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7af86f87-5d83-4a8b-a0cc-88d4641406d2" volumeName="kubernetes.io/projected/7af86f87-5d83-4a8b-a0cc-88d4641406d2-kube-api-access-vq24z" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385527 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e818352-df23-46b4-84d9-81fd259200ec" volumeName="kubernetes.io/projected/8e818352-df23-46b4-84d9-81fd259200ec-kube-api-access-4qplp" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385544 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd6cf358-80f3-42f6-9848-6386a937a57a" volumeName="kubernetes.io/configmap/fd6cf358-80f3-42f6-9848-6386a937a57a-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385557 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="088d4d32-c02b-4da2-9435-c5bc22352079" volumeName="kubernetes.io/projected/088d4d32-c02b-4da2-9435-c5bc22352079-kube-api-access-4t2rp" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385568 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f" volumeName="kubernetes.io/secret/1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f-agent-certs" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385582 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21a54420-0e75-4d4b-93cd-fe72c0cc2070" volumeName="kubernetes.io/secret/21a54420-0e75-4d4b-93cd-fe72c0cc2070-installation-pull-secrets" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385597 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="827bbba0-b306-43db-b805-8c87dfabb874" volumeName="kubernetes.io/projected/827bbba0-b306-43db-b805-8c87dfabb874-kube-api-access-nj9k2" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385609 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95709b19-3c85-4ef1-9110-c23a97847732" volumeName="kubernetes.io/configmap/95709b19-3c85-4ef1-9110-c23a97847732-ovnkube-script-lib" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385622 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9602b28-165a-4c17-96fa-9f0e25fd2cf8" volumeName="kubernetes.io/secret/c9602b28-165a-4c17-96fa-9f0e25fd2cf8-cert" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385648 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e818352-df23-46b4-84d9-81fd259200ec" volumeName="kubernetes.io/empty-dir/8e818352-df23-46b4-84d9-81fd259200ec-tmp-dir" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385661 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="038ae7b2-90e2-44e6-9437-fcb4a0bcfda0" volumeName="kubernetes.io/empty-dir/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-tuned" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385674 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="471ac5c3-1cdd-4d79-bdc1-3acce27cc042" volumeName="kubernetes.io/configmap/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-cni-binary-copy" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385687 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95709b19-3c85-4ef1-9110-c23a97847732" volumeName="kubernetes.io/configmap/95709b19-3c85-4ef1-9110-c23a97847732-ovnkube-config" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385699 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95709b19-3c85-4ef1-9110-c23a97847732" volumeName="kubernetes.io/configmap/95709b19-3c85-4ef1-9110-c23a97847732-env-overrides" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385713 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95709b19-3c85-4ef1-9110-c23a97847732" volumeName="kubernetes.io/secret/95709b19-3c85-4ef1-9110-c23a97847732-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385725 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63" volumeName="kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-kube-api-access-vgxhb" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385737 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63" volumeName="kubernetes.io/secret/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-installation-pull-secrets" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385749 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f" volumeName="kubernetes.io/configmap/1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f-konnectivity-ca" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385762 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="471ac5c3-1cdd-4d79-bdc1-3acce27cc042" volumeName="kubernetes.io/configmap/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-daemon-config" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385775 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4d17d89-b883-472d-84c9-b8f4d0c8eb7e" volumeName="kubernetes.io/configmap/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e-config" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385788 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4d17d89-b883-472d-84c9-b8f4d0c8eb7e" volumeName="kubernetes.io/secret/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e-serving-cert" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385801 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2" volumeName="kubernetes.io/configmap/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-nginx-conf" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385813 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="088d4d32-c02b-4da2-9435-c5bc22352079" volumeName="kubernetes.io/configmap/088d4d32-c02b-4da2-9435-c5bc22352079-serviceca" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385826 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72993a50-bf5d-40c5-833b-afff9f86e61d" volumeName="kubernetes.io/empty-dir/72993a50-bf5d-40c5-833b-afff9f86e61d-data-volume" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385840 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95709b19-3c85-4ef1-9110-c23a97847732" volumeName="kubernetes.io/projected/95709b19-3c85-4ef1-9110-c23a97847732-kube-api-access-dhlzv" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385853 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63" volumeName="kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-bound-sa-token" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385865 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63" volumeName="kubernetes.io/secret/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-image-registry-private-configuration" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385878 4603 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4d17d89-b883-472d-84c9-b8f4d0c8eb7e" volumeName="kubernetes.io/projected/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e-kube-api-access-qbc66" seLinuxMountContext="" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385890 4603 reconstruct.go:97] "Volume reconstruction finished" Oct 30 17:52:19.386176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.385898 4603 reconciler.go:29] "Reconciler: start to sync state" Oct 30 17:52:19.388935 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.388913 4603 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 17:52:19.399306 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.399275 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 17:52:19.400392 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.400367 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 17:52:19.402022 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.401863 4603 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 17:52:19.402022 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.401885 4603 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 17:52:19.402022 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.401906 4603 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 17:52:19.402022 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:19.401946 4603 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 17:52:19.414484 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.414463 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 17:52:19.414870 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.414832 4603 generic.go:356] "Generic (PLEG): container finished" podID="95709b19-3c85-4ef1-9110-c23a97847732" containerID="12088b44ca44e4d059d866fe4f913661c812ad6da59b28ceecc54487e0efdf6c" exitCode=1 Oct 30 17:52:19.416675 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.416650 4603 generic.go:356] "Generic (PLEG): container finished" podID="bb76617dbf2b01fcc9f68e4666f49e08" containerID="692e7ade818756778142f5833e8debdc2d8988dcfa02d3b2f8e27a58b05c0c56" exitCode=0 Oct 30 17:52:19.421771 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.421742 4603 generic.go:356] "Generic (PLEG): container finished" podID="fd6cf358-80f3-42f6-9848-6386a937a57a" containerID="42762e72f271737907c69aa129ffa700e811db4ae9a47b10e2da93fd28aac9c4" exitCode=0 Oct 30 17:52:19.421771 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.421762 4603 generic.go:356] "Generic (PLEG): container finished" podID="fd6cf358-80f3-42f6-9848-6386a937a57a" containerID="1e81aa2a56deaea12669bbd85f3ff99a60dd79569a82838af62f9c7032265de3" exitCode=0 Oct 30 17:52:19.421771 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.421770 4603 generic.go:356] "Generic (PLEG): container finished" podID="fd6cf358-80f3-42f6-9848-6386a937a57a" containerID="4ecd8efb85c6e2ca08221d8959859ac10afcbc615e6cc870a87e578348b00756" exitCode=0 Oct 30 17:52:19.421771 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.421775 4603 generic.go:356] "Generic (PLEG): container finished" podID="fd6cf358-80f3-42f6-9848-6386a937a57a" containerID="2e8440a41a16e5aa29265e6f7118ed5944a7c021e0dd32a24184ba92c5c4050e" exitCode=0 Oct 30 17:52:19.421990 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.421780 4603 generic.go:356] "Generic (PLEG): container finished" podID="fd6cf358-80f3-42f6-9848-6386a937a57a" containerID="1aa632c64fb0b1a8bd30a1fcba31675fa108240401d3604dc4ba351de7431ac5" exitCode=0 Oct 30 17:52:19.421990 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.421785 4603 generic.go:356] "Generic (PLEG): container finished" podID="fd6cf358-80f3-42f6-9848-6386a937a57a" containerID="5ff56b6c3362d55bb9f72d5615a9ab7dafa5de1e2010c0a56e0c97bc6b34016b" exitCode=0 Oct 30 17:52:19.422987 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.422964 4603 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:52:19.423136 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.423121 4603 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 17:52:19.437178 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.437157 4603 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:52:19.437604 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.437574 4603 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:52:19.437708 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.437625 4603 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:52:19.456020 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.456000 4603 manager.go:324] Recovery completed Oct 30 17:52:19.473116 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.473095 4603 cpu_manager.go:226] "Starting" policy="none" Oct 30 17:52:19.473116 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.473112 4603 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 17:52:19.473211 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.473132 4603 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 17:52:19.473291 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.473274 4603 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 17:52:19.473336 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.473315 4603 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 17:52:19.473365 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.473338 4603 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 17:52:19.473365 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.473343 4603 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 17:52:19.473365 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.473349 4603 policy_none.go:50] "Start" Oct 30 17:52:19.473365 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.473363 4603 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 17:52:19.473467 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.473374 4603 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 17:52:19.483027 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.483013 4603 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 17:52:19.483075 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.483030 4603 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 17:52:19.483075 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.483036 4603 policy_none.go:44] "Start" Oct 30 17:52:19.487015 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.486988 4603 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 17:52:19.489582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.489562 4603 manager.go:341] "Starting Device Plugin manager" Oct 30 17:52:19.490124 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:19.489584 4603 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 17:52:19.490124 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.490125 4603 server.go:87] "Starting device plugin registration server" Oct 30 17:52:19.490358 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.490347 4603 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 17:52:19.490390 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.490362 4603 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 17:52:19.490489 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.490459 4603 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 17:52:19.490673 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.490635 4603 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 17:52:19.490747 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.490699 4603 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 17:52:19.490747 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.490707 4603 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 17:52:19.490842 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.490767 4603 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T17:52:19.490687853Z","UUID":"ba522755-968e-4772-a219-dfbf1e259868","Handler":null,"Name":"","Endpoint":""} Oct 30 17:52:19.492014 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:19.491960 4603 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 17:52:19.492170 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.492143 4603 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 17:52:19.492246 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.492174 4603 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 17:52:19.502076 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.502007 4603 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal"] Oct 30 17:52:19.502613 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.502582 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" event={"ID":"bb76617dbf2b01fcc9f68e4666f49e08","Type":"ContainerStarted","Data":"4a295c8c490d1136c64f86524dc2109bd153ae24df1ea79b2edd4c039daae0a1"} Oct 30 17:52:19.502711 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.502625 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" event={"ID":"bb76617dbf2b01fcc9f68e4666f49e08","Type":"ContainerDied","Data":"692e7ade818756778142f5833e8debdc2d8988dcfa02d3b2f8e27a58b05c0c56"} Oct 30 17:52:19.502711 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.502649 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" event={"ID":"bb76617dbf2b01fcc9f68e4666f49e08","Type":"ContainerStarted","Data":"bd5a6a5bef5bd27b9d1982aabd2157ac61ec13df9888f62fb9027c666fcfbe0e"} Oct 30 17:52:19.502711 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.502597 4603 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" Oct 30 17:52:19.502711 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.502702 4603 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal" Oct 30 17:52:19.502867 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.502694 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal" event={"ID":"f50f91e38dadbc11b70b476e358bc37f","Type":"ContainerStarted","Data":"c472835fdfec63debb2385e7e792c37045717452ab7cdf7fe0b8384c78fc6fa5"} Oct 30 17:52:19.502867 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.502854 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal" event={"ID":"f50f91e38dadbc11b70b476e358bc37f","Type":"ContainerStarted","Data":"81c432afc2475543bfeda559a25113dc69a47425c99a1a739b7416bf9acf983d"} Oct 30 17:52:19.532024 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.532004 4603 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:52:19.532076 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:19.532045 4603 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-236.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal" Oct 30 17:52:19.532559 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.532547 4603 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:52:19.532601 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:19.532577 4603 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" Oct 30 17:52:19.591284 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.591267 4603 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:52:19.592307 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.592292 4603 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:52:19.592382 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.592317 4603 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:52:19.592382 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.592326 4603 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:52:19.592382 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.592358 4603 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-236.ec2.internal" Oct 30 17:52:19.624587 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.624567 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb76617dbf2b01fcc9f68e4666f49e08-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal\" (UID: \"bb76617dbf2b01fcc9f68e4666f49e08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" Oct 30 17:52:19.624672 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.624591 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb76617dbf2b01fcc9f68e4666f49e08-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal\" (UID: \"bb76617dbf2b01fcc9f68e4666f49e08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" Oct 30 17:52:19.624672 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.624608 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f50f91e38dadbc11b70b476e358bc37f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-236.ec2.internal\" (UID: \"f50f91e38dadbc11b70b476e358bc37f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal" Oct 30 17:52:19.626110 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.626097 4603 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-236.ec2.internal" Oct 30 17:52:19.626160 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.626144 4603 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-236.ec2.internal" Oct 30 17:52:19.725351 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.725333 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f50f91e38dadbc11b70b476e358bc37f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-236.ec2.internal\" (UID: \"f50f91e38dadbc11b70b476e358bc37f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal" Oct 30 17:52:19.725441 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.725357 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb76617dbf2b01fcc9f68e4666f49e08-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal\" (UID: \"bb76617dbf2b01fcc9f68e4666f49e08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" Oct 30 17:52:19.725441 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.725373 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb76617dbf2b01fcc9f68e4666f49e08-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal\" (UID: \"bb76617dbf2b01fcc9f68e4666f49e08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" Oct 30 17:52:19.725441 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.725407 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb76617dbf2b01fcc9f68e4666f49e08-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal\" (UID: \"bb76617dbf2b01fcc9f68e4666f49e08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" Oct 30 17:52:19.725568 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.725440 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f50f91e38dadbc11b70b476e358bc37f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-236.ec2.internal\" (UID: \"f50f91e38dadbc11b70b476e358bc37f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal" Oct 30 17:52:19.725568 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:19.725453 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb76617dbf2b01fcc9f68e4666f49e08-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal\" (UID: \"bb76617dbf2b01fcc9f68e4666f49e08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" Oct 30 17:52:20.360491 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.360453 4603 apiserver.go:52] "Watching apiserver" Oct 30 17:52:20.399056 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.399026 4603 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 17:52:20.399690 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.399673 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7txkr","openshift-dns/node-resolver-pbqhh","kube-system/global-pull-secret-syncer-5drgz","openshift-network-operator/iptables-alerter-db4lm","openshift-ovn-kubernetes/ovnkube-node-7cgxg","openshift-dns/dns-default-7r5h8","openshift-kube-storage-version-migrator/migrator-66df5d496b-jr2r5","openshift-image-registry/node-ca-4hx5j","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7","openshift-multus/network-metrics-daemon-jnxk7","kube-system/konnectivity-agent-pgghg","openshift-image-registry/image-registry-69885b9d-n2bmz","openshift-multus/multus-additional-cni-plugins-v5l7g","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal","openshift-network-diagnostics/network-check-source-6f78d59f99-w725f","openshift-network-diagnostics/network-check-target-4xxmr","openshift-image-registry/image-registry-64bf8b8bb7-dlchr","openshift-insights/insights-runtime-extractor-nsw4r","openshift-network-console/networking-console-plugin-86c478bb88-kkgjk","kube-system/kas-connection-checker-685895c7c9-6xn6r","openshift-multus/multus-th8wr","kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal","openshift-cluster-node-tuning-operator/tuned-khz68"] Oct 30 17:52:20.401000 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.400967 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:20.402959 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.402938 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-kkgjk" Oct 30 17:52:20.405074 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.405057 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:52:20.408798 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.408782 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:20.409149 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.409138 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:20.410516 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.410487 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:20.414685 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.414665 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 17:52:20.414759 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.414691 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 17:52:20.414759 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.414744 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 17:52:20.415153 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.415138 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 17:52:20.415249 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.415237 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:52:20.415406 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.415391 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 17:52:20.415839 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.415825 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 17:52:20.415929 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.415915 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 17:52:20.416155 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.416125 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 17:52:20.416413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.416394 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 17:52:20.416478 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.416434 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-96ztk\"" Oct 30 17:52:20.416622 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.416610 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 17:52:20.416679 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.416668 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vwtrh\"" Oct 30 17:52:20.427890 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.427874 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 17:52:20.427957 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.427887 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-wfmdw\"" Oct 30 17:52:20.428136 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.428123 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 17:52:20.432357 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.432342 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 17:52:20.432495 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.432482 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 17:52:20.432564 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.432481 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 17:52:20.432564 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.432556 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-twdt2\"" Oct 30 17:52:20.432663 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.432621 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 17:52:20.432663 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.432629 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 17:52:20.432663 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.432657 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-zd7pm\"" Oct 30 17:52:20.432784 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.432704 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fzft4\"" Oct 30 17:52:20.432784 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.432766 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 17:52:20.435561 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.435545 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 17:52:20.435685 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.435667 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 17:52:20.435796 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.435702 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 17:52:20.436059 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.436044 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:52:20.436125 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.436064 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 17:52:20.436125 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.436118 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 17:52:20.436229 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.436124 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 17:52:20.436229 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.436140 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 17:52:20.436229 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.436194 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 17:52:20.436229 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.436203 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 17:52:20.436229 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.436214 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 17:52:20.436389 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.436241 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 17:52:20.436389 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.436205 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 17:52:20.436749 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.436724 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 17:52:20.436849 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.436834 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 17:52:20.436920 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.436844 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 17:52:20.436920 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.436854 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:52:20.437078 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.437062 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 17:52:20.437130 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.437064 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 17:52:20.437201 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.437183 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 17:52:20.437405 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.437388 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 17:52:20.437459 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.437426 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 17:52:20.437538 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.437379 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 17:52:20.437583 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.437550 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 17:52:20.437654 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.437636 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 17:52:20.437820 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.437800 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-99rd9\"" Oct 30 17:52:20.437900 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.437883 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 17:52:20.438036 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.438018 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-k5mms\"" Oct 30 17:52:20.438115 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.438025 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 17:52:20.438170 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.438147 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 17:52:20.438170 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.437884 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 17:52:20.438263 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.438201 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ddhr5\"" Oct 30 17:52:20.438860 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.438842 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k56gt\"" Oct 30 17:52:20.438937 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.438917 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zlfnf\"" Oct 30 17:52:20.438990 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.438953 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cxbhw\"" Oct 30 17:52:20.439047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.439032 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5kfdv\"" Oct 30 17:52:20.439151 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.439135 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 17:52:20.439215 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.439136 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6xmc6\"" Oct 30 17:52:20.439215 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.439194 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 17:52:20.439305 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.439256 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 17:52:20.439360 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.439343 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-cgqqv\"" Oct 30 17:52:20.439416 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.439374 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 17:52:20.439459 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.439378 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 17:52:20.439459 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.439405 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hhs6k\"" Oct 30 17:52:20.439609 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.439577 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lsd9l\"" Oct 30 17:52:20.439698 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.439625 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nb8hn\"" Oct 30 17:52:20.439698 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.439634 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 17:52:20.439806 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.439790 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dqbz4\"" Oct 30 17:52:20.440128 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.440113 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-x8qzn\"" Oct 30 17:52:20.447073 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.447055 4603 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal" Oct 30 17:52:20.447185 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.447134 4603 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" Oct 30 17:52:20.473451 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.473424 4603 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:52:20.473557 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:20.473474 4603 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-236.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal" Oct 30 17:52:20.473557 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.473534 4603 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:52:20.473658 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:20.473575 4603 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" Oct 30 17:52:20.478778 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.478763 4603 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 17:52:20.530785 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.530764 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-host\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.530871 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.530795 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:20.530871 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.530826 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qplp\" (UniqueName: \"kubernetes.io/projected/8e818352-df23-46b4-84d9-81fd259200ec-kube-api-access-4qplp\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:20.530976 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.530879 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t2rp\" (UniqueName: \"kubernetes.io/projected/088d4d32-c02b-4da2-9435-c5bc22352079-kube-api-access-4t2rp\") pod \"node-ca-4hx5j\" (UID: \"088d4d32-c02b-4da2-9435-c5bc22352079\") " pod="openshift-image-registry/node-ca-4hx5j" Oct 30 17:52:20.530976 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.530951 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-cni-binary-copy\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.531073 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531056 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-etc-openvswitch\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.531122 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531075 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.531122 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531110 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf\" (UID: \"b4d17d89-b883-472d-84c9-b8f4d0c8eb7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf" Oct 30 17:52:20.531213 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531128 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21a54420-0e75-4d4b-93cd-fe72c0cc2070-ca-trust-extracted\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:20.531213 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531165 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21a54420-0e75-4d4b-93cd-fe72c0cc2070-installation-pull-secrets\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:20.531213 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531189 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21a54420-0e75-4d4b-93cd-fe72c0cc2070-ca-trust-extracted\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:20.531213 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531188 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-run-ovn-kubernetes\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.531383 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531221 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhlzv\" (UniqueName: \"kubernetes.io/projected/95709b19-3c85-4ef1-9110-c23a97847732-kube-api-access-dhlzv\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.531383 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531269 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-cni-binary-copy\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.531383 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531293 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf\" (UID: \"b4d17d89-b883-472d-84c9-b8f4d0c8eb7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf" Oct 30 17:52:20.531383 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531313 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kkgjk\" (UID: \"f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkgjk" Oct 30 17:52:20.531383 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531345 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-hostroot\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.531383 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531376 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs\") pod \"network-metrics-daemon-jnxk7\" (UID: \"827bbba0-b306-43db-b805-8c87dfabb874\") " pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:52:20.531612 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531396 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:52:20.531612 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531410 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21a54420-0e75-4d4b-93cd-fe72c0cc2070-installation-pull-secrets\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:20.531612 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531414 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:52:20.531612 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531458 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-sysctl-conf\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.531612 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531490 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbs6z\" (UniqueName: \"kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z\") pod \"network-check-target-4xxmr\" (UID: \"90437f91-1657-47db-b3a1-d59bc8d1de14\") " pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:52:20.531612 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531539 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-var-lib-cni-bin\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.531612 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531543 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kkgjk\" (UID: \"f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkgjk" Oct 30 17:52:20.531612 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531562 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ks5l\" (UniqueName: \"kubernetes.io/projected/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-kube-api-access-8ks5l\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.531612 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531587 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fd6cf358-80f3-42f6-9848-6386a937a57a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:52:20.531976 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531697 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-node-log\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.531976 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531733 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/95709b19-3c85-4ef1-9110-c23a97847732-ovnkube-script-lib\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.531976 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531764 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj9k2\" (UniqueName: \"kubernetes.io/projected/827bbba0-b306-43db-b805-8c87dfabb874-kube-api-access-nj9k2\") pod \"network-metrics-daemon-jnxk7\" (UID: \"827bbba0-b306-43db-b805-8c87dfabb874\") " pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:52:20.531976 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531791 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-modprobe-d\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.531976 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531821 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgxhb\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-kube-api-access-vgxhb\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:20.531976 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531849 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxtq4\" (UniqueName: \"kubernetes.io/projected/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-api-access-lxtq4\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:20.531976 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531895 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21a54420-0e75-4d4b-93cd-fe72c0cc2070-trusted-ca\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:20.531976 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531896 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fd6cf358-80f3-42f6-9848-6386a937a57a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:52:20.531976 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531930 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-cni-dir\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.531976 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531968 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-run-openvswitch\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.532421 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.531986 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/95709b19-3c85-4ef1-9110-c23a97847732-ovnkube-script-lib\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.532421 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532026 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret\") pod \"global-pull-secret-syncer-5drgz\" (UID: \"1e6590e8-bfc4-4b19-b579-7180aa6af22a\") " pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:52:20.532421 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532064 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8e818352-df23-46b4-84d9-81fd259200ec-config-volume\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:20.532421 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532182 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lnnm\" (UniqueName: \"kubernetes.io/projected/e9933e90-d394-4e89-8e0f-115298c359fa-kube-api-access-2lnnm\") pod \"iptables-alerter-db4lm\" (UID: \"e9933e90-d394-4e89-8e0f-115298c359fa\") " pod="openshift-network-operator/iptables-alerter-db4lm" Oct 30 17:52:20.532421 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532214 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e6590e8-bfc4-4b19-b579-7180aa6af22a-original-pull-secret\") pod \"global-pull-secret-syncer-5drgz\" (UID: \"1e6590e8-bfc4-4b19-b579-7180aa6af22a\") " pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:52:20.532421 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532215 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-socket-dir-parent\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.532421 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532297 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8e818352-df23-46b4-84d9-81fd259200ec-config-volume\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:20.532421 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532314 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-systemd\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.532421 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532316 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21a54420-0e75-4d4b-93cd-fe72c0cc2070-trusted-ca\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:20.532421 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532336 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-system-cni-dir\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.532421 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532357 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-installation-pull-secrets\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:20.532421 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532375 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk78z\" (UniqueName: \"kubernetes.io/projected/fd6cf358-80f3-42f6-9848-6386a937a57a-kube-api-access-pk78z\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:52:20.532421 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532394 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/088d4d32-c02b-4da2-9435-c5bc22352079-serviceca\") pod \"node-ca-4hx5j\" (UID: \"088d4d32-c02b-4da2-9435-c5bc22352079\") " pod="openshift-image-registry/node-ca-4hx5j" Oct 30 17:52:20.532421 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532414 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-log-socket\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.532966 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532438 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkgjk\" (UID: \"f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkgjk" Oct 30 17:52:20.532966 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532455 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-sysctl-d\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.532966 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532482 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-bound-sa-token\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:20.532966 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532491 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-installation-pull-secrets\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:20.532966 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532518 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72993a50-bf5d-40c5-833b-afff9f86e61d-crio-socket\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:20.532966 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532547 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72993a50-bf5d-40c5-833b-afff9f86e61d-data-volume\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:20.532966 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532611 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-lib-modules\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.532966 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532613 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72993a50-bf5d-40c5-833b-afff9f86e61d-data-volume\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:20.532966 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532648 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9602b28-165a-4c17-96fa-9f0e25fd2cf8-cert\") pod \"ingress-canary-7txkr\" (UID: \"c9602b28-165a-4c17-96fa-9f0e25fd2cf8\") " pod="openshift-ingress-canary/ingress-canary-7txkr" Oct 30 17:52:20.532966 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532671 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-run\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.532966 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532687 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzf62\" (UniqueName: \"kubernetes.io/projected/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-kube-api-access-wzf62\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.532966 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532705 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-sys-fs\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:52:20.532966 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532714 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/088d4d32-c02b-4da2-9435-c5bc22352079-serviceca\") pod \"node-ca-4hx5j\" (UID: \"088d4d32-c02b-4da2-9435-c5bc22352079\") " pod="openshift-image-registry/node-ca-4hx5j" Oct 30 17:52:20.532966 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532808 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fd6cf358-80f3-42f6-9848-6386a937a57a-cni-binary-copy\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:52:20.532966 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532831 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9602b28-165a-4c17-96fa-9f0e25fd2cf8-cert\") pod \"ingress-canary-7txkr\" (UID: \"c9602b28-165a-4c17-96fa-9f0e25fd2cf8\") " pod="openshift-ingress-canary/ingress-canary-7txkr" Oct 30 17:52:20.532966 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532836 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-tls\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:20.532966 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532855 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-var-lib-cni-multus\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.532966 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532892 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-conf-dir\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.532966 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532915 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-ca-trust-extracted\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:20.532966 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532931 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9933e90-d394-4e89-8e0f-115298c359fa-host-slash\") pod \"iptables-alerter-db4lm\" (UID: \"e9933e90-d394-4e89-8e0f-115298c359fa\") " pod="openshift-network-operator/iptables-alerter-db4lm" Oct 30 17:52:20.532966 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532964 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-cnibin\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.532966 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.532967 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fd6cf358-80f3-42f6-9848-6386a937a57a-cni-binary-copy\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:52:20.533639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533002 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-ca-trust-extracted\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:20.533639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533034 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e9933e90-d394-4e89-8e0f-115298c359fa-iptables-alerter-script\") pod \"iptables-alerter-db4lm\" (UID: \"e9933e90-d394-4e89-8e0f-115298c359fa\") " pod="openshift-network-operator/iptables-alerter-db4lm" Oct 30 17:52:20.533639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533055 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-var-lib-kubelet\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.533639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533079 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-kubernetes\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.533639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533169 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-registration-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:52:20.533639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533196 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd6cf358-80f3-42f6-9848-6386a937a57a-cnibin\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:52:20.533639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533215 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-etc-kubernetes\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.533639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533241 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-tmp\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.533639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533249 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e9933e90-d394-4e89-8e0f-115298c359fa-iptables-alerter-script\") pod \"iptables-alerter-db4lm\" (UID: \"e9933e90-d394-4e89-8e0f-115298c359fa\") " pod="openshift-network-operator/iptables-alerter-db4lm" Oct 30 17:52:20.533639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533264 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-certificates\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:20.533639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533297 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-tmp\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.533639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533368 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-certificates\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:20.533639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533390 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-run-k8s-cni-cncf-io\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.533639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533414 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-kubelet\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.533639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533441 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbc66\" (UniqueName: \"kubernetes.io/projected/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e-kube-api-access-qbc66\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf\" (UID: \"b4d17d89-b883-472d-84c9-b8f4d0c8eb7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf" Oct 30 17:52:20.533639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533474 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9xxt\" (UniqueName: \"kubernetes.io/projected/c9602b28-165a-4c17-96fa-9f0e25fd2cf8-kube-api-access-r9xxt\") pod \"ingress-canary-7txkr\" (UID: \"c9602b28-165a-4c17-96fa-9f0e25fd2cf8\") " pod="openshift-ingress-canary/ingress-canary-7txkr" Oct 30 17:52:20.533639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533516 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-var-lib-kubelet\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.533639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533614 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf\" (UID: \"b4d17d89-b883-472d-84c9-b8f4d0c8eb7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf" Oct 30 17:52:20.533639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533638 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-tls\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:20.534177 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533730 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3ef04039-ef85-40ef-8f30-a0edd7b402e0-hosts-file\") pod \"node-resolver-pbqhh\" (UID: \"3ef04039-ef85-40ef-8f30-a0edd7b402e0\") " pod="openshift-dns/node-resolver-pbqhh" Oct 30 17:52:20.534177 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533756 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-systemd-units\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.534177 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533762 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf\" (UID: \"b4d17d89-b883-472d-84c9-b8f4d0c8eb7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf" Oct 30 17:52:20.534177 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533779 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-cni-netd\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.534177 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533796 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4bqp\" (UniqueName: \"kubernetes.io/projected/355e77f7-f944-4955-95c2-bdf4c7ea12b7-kube-api-access-d4bqp\") pod \"kas-connection-checker-685895c7c9-6xn6r\" (UID: \"355e77f7-f944-4955-95c2-bdf4c7ea12b7\") " pod="kube-system/kas-connection-checker-685895c7c9-6xn6r" Oct 30 17:52:20.534177 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533814 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-image-registry-private-configuration\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:20.534177 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533914 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f-konnectivity-ca\") pod \"konnectivity-agent-pgghg\" (UID: \"1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f\") " pod="kube-system/konnectivity-agent-pgghg" Oct 30 17:52:20.534177 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533949 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/95709b19-3c85-4ef1-9110-c23a97847732-ovn-node-metrics-cert\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.534177 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.533981 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-device-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:52:20.534177 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534017 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm6vb\" (UniqueName: \"kubernetes.io/projected/3ef04039-ef85-40ef-8f30-a0edd7b402e0-kube-api-access-lm6vb\") pod \"node-resolver-pbqhh\" (UID: \"3ef04039-ef85-40ef-8f30-a0edd7b402e0\") " pod="openshift-dns/node-resolver-pbqhh" Oct 30 17:52:20.534177 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534041 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f-agent-certs\") pod \"konnectivity-agent-pgghg\" (UID: \"1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f\") " pod="kube-system/konnectivity-agent-pgghg" Oct 30 17:52:20.534177 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534082 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-image-registry-private-configuration\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:20.534177 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534099 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f-konnectivity-ca\") pod \"konnectivity-agent-pgghg\" (UID: \"1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f\") " pod="kube-system/konnectivity-agent-pgghg" Oct 30 17:52:20.534177 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534098 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-run-systemd\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.534726 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534204 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd6cf358-80f3-42f6-9848-6386a937a57a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:52:20.534726 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534205 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/95709b19-3c85-4ef1-9110-c23a97847732-ovn-node-metrics-cert\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.534726 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534285 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f-agent-certs\") pod \"konnectivity-agent-pgghg\" (UID: \"1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f\") " pod="kube-system/konnectivity-agent-pgghg" Oct 30 17:52:20.534726 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534294 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef04039-ef85-40ef-8f30-a0edd7b402e0-tmp-dir\") pod \"node-resolver-pbqhh\" (UID: \"3ef04039-ef85-40ef-8f30-a0edd7b402e0\") " pod="openshift-dns/node-resolver-pbqhh" Oct 30 17:52:20.534726 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534328 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-slash\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.534726 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534345 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef04039-ef85-40ef-8f30-a0edd7b402e0-tmp-dir\") pod \"node-resolver-pbqhh\" (UID: \"3ef04039-ef85-40ef-8f30-a0edd7b402e0\") " pod="openshift-dns/node-resolver-pbqhh" Oct 30 17:52:20.534726 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534357 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/95709b19-3c85-4ef1-9110-c23a97847732-ovnkube-config\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.534726 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534385 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:20.534726 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534416 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-bound-sa-token\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:20.534726 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534441 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-run-ovn\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.534726 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534466 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e6590e8-bfc4-4b19-b579-7180aa6af22a-kubelet-config\") pod \"global-pull-secret-syncer-5drgz\" (UID: \"1e6590e8-bfc4-4b19-b579-7180aa6af22a\") " pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:52:20.534726 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534495 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/95709b19-3c85-4ef1-9110-c23a97847732-ovnkube-config\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.534726 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534541 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-sysconfig\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.534726 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534560 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd6cf358-80f3-42f6-9848-6386a937a57a-system-cni-dir\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:52:20.534726 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534583 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:20.534726 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534609 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21a54420-0e75-4d4b-93cd-fe72c0cc2070-image-registry-private-configuration\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:20.534726 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534712 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-run-multus-certs\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.535413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534741 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-run-netns\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.535413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534731 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21a54420-0e75-4d4b-93cd-fe72c0cc2070-image-registry-private-configuration\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:20.535413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534761 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-cni-bin\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.535413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534796 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-sys\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.535413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534824 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq24z\" (UniqueName: \"kubernetes.io/projected/7af86f87-5d83-4a8b-a0cc-88d4641406d2-kube-api-access-vq24z\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:52:20.535413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534840 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z85fd\" (UniqueName: \"kubernetes.io/projected/1a6fe435-1778-45b0-ae18-6695f5824229-kube-api-access-z85fd\") pod \"migrator-66df5d496b-jr2r5\" (UID: \"1a6fe435-1778-45b0-ae18-6695f5824229\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jr2r5" Oct 30 17:52:20.535413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534861 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/95709b19-3c85-4ef1-9110-c23a97847732-env-overrides\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.535413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534945 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-tuned\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.535413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.534977 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-socket-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:52:20.535413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.535019 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/95709b19-3c85-4ef1-9110-c23a97847732-env-overrides\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.535413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.535035 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-tuned\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.535413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.535048 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e6590e8-bfc4-4b19-b579-7180aa6af22a-dbus\") pod \"global-pull-secret-syncer-5drgz\" (UID: \"1e6590e8-bfc4-4b19-b579-7180aa6af22a\") " pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:52:20.535413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.535094 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-certificates\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:20.535413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.535129 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmwhl\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-kube-api-access-hmwhl\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:20.535413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.535185 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-daemon-config\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.535413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.535229 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-certificates\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:20.535413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.535256 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-var-lib-openvswitch\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.535413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.535292 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-trusted-ca\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:20.535413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.535309 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll92d\" (UniqueName: \"kubernetes.io/projected/d58a06a1-fdbc-439e-8572-fdeabc6ee4ef-kube-api-access-ll92d\") pod \"network-check-source-6f78d59f99-w725f\" (UID: \"d58a06a1-fdbc-439e-8572-fdeabc6ee4ef\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-w725f" Oct 30 17:52:20.535413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.535352 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/088d4d32-c02b-4da2-9435-c5bc22352079-host\") pod \"node-ca-4hx5j\" (UID: \"088d4d32-c02b-4da2-9435-c5bc22352079\") " pod="openshift-image-registry/node-ca-4hx5j" Oct 30 17:52:20.535413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.535357 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-daemon-config\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.535979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.535461 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-run-netns\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.535979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.535488 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fd6cf358-80f3-42f6-9848-6386a937a57a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:52:20.535979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.535532 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e818352-df23-46b4-84d9-81fd259200ec-tmp-dir\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:20.535979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.535581 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e818352-df23-46b4-84d9-81fd259200ec-tmp-dir\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:20.535979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.535694 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-trusted-ca\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:20.535979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.535708 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fd6cf358-80f3-42f6-9848-6386a937a57a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:52:20.554257 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.554238 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qplp\" (UniqueName: \"kubernetes.io/projected/8e818352-df23-46b4-84d9-81fd259200ec-kube-api-access-4qplp\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:20.554415 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.554400 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhlzv\" (UniqueName: \"kubernetes.io/projected/95709b19-3c85-4ef1-9110-c23a97847732-kube-api-access-dhlzv\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.564695 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.564678 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmwhl\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-kube-api-access-hmwhl\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:20.565029 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.565010 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxtq4\" (UniqueName: \"kubernetes.io/projected/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-api-access-lxtq4\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:20.565109 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.565094 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4bqp\" (UniqueName: \"kubernetes.io/projected/355e77f7-f944-4955-95c2-bdf4c7ea12b7-kube-api-access-d4bqp\") pod \"kas-connection-checker-685895c7c9-6xn6r\" (UID: \"355e77f7-f944-4955-95c2-bdf4c7ea12b7\") " pod="kube-system/kas-connection-checker-685895c7c9-6xn6r" Oct 30 17:52:20.565139 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.565108 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbc66\" (UniqueName: \"kubernetes.io/projected/b4d17d89-b883-472d-84c9-b8f4d0c8eb7e-kube-api-access-qbc66\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf\" (UID: \"b4d17d89-b883-472d-84c9-b8f4d0c8eb7e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf" Oct 30 17:52:20.565194 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.565179 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t2rp\" (UniqueName: \"kubernetes.io/projected/088d4d32-c02b-4da2-9435-c5bc22352079-kube-api-access-4t2rp\") pod \"node-ca-4hx5j\" (UID: \"088d4d32-c02b-4da2-9435-c5bc22352079\") " pod="openshift-image-registry/node-ca-4hx5j" Oct 30 17:52:20.566992 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.566977 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj9k2\" (UniqueName: \"kubernetes.io/projected/827bbba0-b306-43db-b805-8c87dfabb874-kube-api-access-nj9k2\") pod \"network-metrics-daemon-jnxk7\" (UID: \"827bbba0-b306-43db-b805-8c87dfabb874\") " pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:52:20.572186 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.572166 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-bound-sa-token\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:20.572372 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.572358 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ks5l\" (UniqueName: \"kubernetes.io/projected/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-kube-api-access-8ks5l\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.572438 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.572422 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9xxt\" (UniqueName: \"kubernetes.io/projected/c9602b28-165a-4c17-96fa-9f0e25fd2cf8-kube-api-access-r9xxt\") pod \"ingress-canary-7txkr\" (UID: \"c9602b28-165a-4c17-96fa-9f0e25fd2cf8\") " pod="openshift-ingress-canary/ingress-canary-7txkr" Oct 30 17:52:20.572475 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.572447 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-bound-sa-token\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:20.572755 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.572741 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgxhb\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-kube-api-access-vgxhb\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:20.573049 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.573035 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm6vb\" (UniqueName: \"kubernetes.io/projected/3ef04039-ef85-40ef-8f30-a0edd7b402e0-kube-api-access-lm6vb\") pod \"node-resolver-pbqhh\" (UID: \"3ef04039-ef85-40ef-8f30-a0edd7b402e0\") " pod="openshift-dns/node-resolver-pbqhh" Oct 30 17:52:20.573545 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.573531 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lnnm\" (UniqueName: \"kubernetes.io/projected/e9933e90-d394-4e89-8e0f-115298c359fa-kube-api-access-2lnnm\") pod \"iptables-alerter-db4lm\" (UID: \"e9933e90-d394-4e89-8e0f-115298c359fa\") " pod="openshift-network-operator/iptables-alerter-db4lm" Oct 30 17:52:20.573742 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.573728 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z85fd\" (UniqueName: \"kubernetes.io/projected/1a6fe435-1778-45b0-ae18-6695f5824229-kube-api-access-z85fd\") pod \"migrator-66df5d496b-jr2r5\" (UID: \"1a6fe435-1778-45b0-ae18-6695f5824229\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jr2r5" Oct 30 17:52:20.577702 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.577688 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk78z\" (UniqueName: \"kubernetes.io/projected/fd6cf358-80f3-42f6-9848-6386a937a57a-kube-api-access-pk78z\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:52:20.578429 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.578407 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll92d\" (UniqueName: \"kubernetes.io/projected/d58a06a1-fdbc-439e-8572-fdeabc6ee4ef-kube-api-access-ll92d\") pod \"network-check-source-6f78d59f99-w725f\" (UID: \"d58a06a1-fdbc-439e-8572-fdeabc6ee4ef\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-w725f" Oct 30 17:52:20.578568 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.578545 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzf62\" (UniqueName: \"kubernetes.io/projected/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-kube-api-access-wzf62\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.578791 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.578772 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq24z\" (UniqueName: \"kubernetes.io/projected/7af86f87-5d83-4a8b-a0cc-88d4641406d2-kube-api-access-vq24z\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:52:20.579101 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.579078 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbs6z\" (UniqueName: \"kubernetes.io/projected/90437f91-1657-47db-b3a1-d59bc8d1de14-kube-api-access-fbs6z\") pod \"network-check-target-4xxmr\" (UID: \"90437f91-1657-47db-b3a1-d59bc8d1de14\") " pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:52:20.579492 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.579474 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4hx5j_088d4d32-c02b-4da2-9435-c5bc22352079/node-ca/0.log" Oct 30 17:52:20.609727 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.609697 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:20.636833 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.636791 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-run-k8s-cni-cncf-io\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.636833 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.636813 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-kubelet\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.636967 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.636836 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-var-lib-kubelet\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.636967 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.636852 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-tls\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:20.636967 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.636869 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3ef04039-ef85-40ef-8f30-a0edd7b402e0-hosts-file\") pod \"node-resolver-pbqhh\" (UID: \"3ef04039-ef85-40ef-8f30-a0edd7b402e0\") " pod="openshift-dns/node-resolver-pbqhh" Oct 30 17:52:20.636967 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.636881 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-systemd-units\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.636967 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.636900 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-cni-netd\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.636967 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.636904 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-run-k8s-cni-cncf-io\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.636967 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.636927 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-device-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:52:20.636967 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.636900 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-kubelet\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.636967 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.636944 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-var-lib-kubelet\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.636967 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.636953 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-run-systemd\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.636967 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.636954 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-systemd-units\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.636967 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.636945 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3ef04039-ef85-40ef-8f30-a0edd7b402e0-hosts-file\") pod \"node-resolver-pbqhh\" (UID: \"3ef04039-ef85-40ef-8f30-a0edd7b402e0\") " pod="openshift-dns/node-resolver-pbqhh" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637000 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-run-systemd\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637001 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-device-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637024 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd6cf358-80f3-42f6-9848-6386a937a57a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.636998 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-cni-netd\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637048 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-slash\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637054 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd6cf358-80f3-42f6-9848-6386a937a57a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637077 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637085 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-slash\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637104 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-run-ovn\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637125 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e6590e8-bfc4-4b19-b579-7180aa6af22a-kubelet-config\") pod \"global-pull-secret-syncer-5drgz\" (UID: \"1e6590e8-bfc4-4b19-b579-7180aa6af22a\") " pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637139 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-sysconfig\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637163 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd6cf358-80f3-42f6-9848-6386a937a57a-system-cni-dir\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:20.637189 4603 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637180 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637200 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd6cf358-80f3-42f6-9848-6386a937a57a-system-cni-dir\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637198 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-sysconfig\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637162 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-run-ovn\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637191 4603 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637192 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e6590e8-bfc4-4b19-b579-7180aa6af22a-kubelet-config\") pod \"global-pull-secret-syncer-5drgz\" (UID: \"1e6590e8-bfc4-4b19-b579-7180aa6af22a\") " pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637237 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-run-multus-certs\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:20.637276 4603 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls podName:72993a50-bf5d-40c5-833b-afff9f86e61d nodeName:}" failed. No retries permitted until 2026-10-30 17:52:21.137254931 +0000 UTC m=+1.852824806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nsw4r" (UID: "72993a50-bf5d-40c5-833b-afff9f86e61d") : secret "insights-runtime-extractor-tls" not found Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637278 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-run-multus-certs\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:20.637283 4603 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637294 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-run-netns\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637324 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-cni-bin\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637333 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-run-netns\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:20.637346 4603 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls podName:8e818352-df23-46b4-84d9-81fd259200ec nodeName:}" failed. No retries permitted until 2026-10-30 17:52:21.137329705 +0000 UTC m=+1.852899579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls") pod "dns-default-7r5h8" (UID: "8e818352-df23-46b4-84d9-81fd259200ec") : secret "dns-default-metrics-tls" not found Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637345 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-cni-bin\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637379 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-sys\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637408 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-socket-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637432 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e6590e8-bfc4-4b19-b579-7180aa6af22a-dbus\") pod \"global-pull-secret-syncer-5drgz\" (UID: \"1e6590e8-bfc4-4b19-b579-7180aa6af22a\") " pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637448 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-sys\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637457 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-var-lib-openvswitch\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637481 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/088d4d32-c02b-4da2-9435-c5bc22352079-host\") pod \"node-ca-4hx5j\" (UID: \"088d4d32-c02b-4da2-9435-c5bc22352079\") " pod="openshift-image-registry/node-ca-4hx5j" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637519 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-socket-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637524 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e6590e8-bfc4-4b19-b579-7180aa6af22a-dbus\") pod \"global-pull-secret-syncer-5drgz\" (UID: \"1e6590e8-bfc4-4b19-b579-7180aa6af22a\") " pod="kube-system/global-pull-secret-syncer-5drgz" Oct 30 17:52:20.637493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637524 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-run-netns\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637560 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-host\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637563 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/088d4d32-c02b-4da2-9435-c5bc22352079-host\") pod \"node-ca-4hx5j\" (UID: \"088d4d32-c02b-4da2-9435-c5bc22352079\") " pod="openshift-image-registry/node-ca-4hx5j" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637565 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-run-netns\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637612 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-host\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637627 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-var-lib-openvswitch\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637633 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:20.637672 4603 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637676 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-etc-openvswitch\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:20.637709 4603 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-rbac-proxy-cm podName:72993a50-bf5d-40c5-833b-afff9f86e61d nodeName:}" failed. No retries permitted until 2026-10-30 17:52:21.137700526 +0000 UTC m=+1.853270398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nsw4r" (UID: "72993a50-bf5d-40c5-833b-afff9f86e61d") : configmap "kube-rbac-proxy" not found Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637704 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637729 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-run-ovn-kubernetes\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637739 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637745 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-hostroot\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637763 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-etc-openvswitch\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637771 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-hostroot\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637780 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-host-run-ovn-kubernetes\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637782 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs\") pod \"network-metrics-daemon-jnxk7\" (UID: \"827bbba0-b306-43db-b805-8c87dfabb874\") " pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637810 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637828 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:20.637840 4603 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637849 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-sysctl-conf\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637873 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637883 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:20.637880 4603 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs podName:827bbba0-b306-43db-b805-8c87dfabb874 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:21.137866156 +0000 UTC m=+1.853436031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs") pod "network-metrics-daemon-jnxk7" (UID: "827bbba0-b306-43db-b805-8c87dfabb874") : secret "metrics-daemon-secret" not found Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637912 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-var-lib-cni-bin\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637933 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-node-log\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637933 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-var-lib-cni-bin\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637954 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-node-log\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637969 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-sysctl-conf\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.637955 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-modprobe-d\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638010 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-cni-dir\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638031 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-run-openvswitch\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638048 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-modprobe-d\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638055 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-socket-dir-parent\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638060 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-cni-dir\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638078 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-systemd\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638086 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-run-openvswitch\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638103 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-system-cni-dir\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638130 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-log-socket\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638145 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-systemd\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638151 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkgjk\" (UID: \"f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkgjk" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638174 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95709b19-3c85-4ef1-9110-c23a97847732-log-socket\") pod \"ovnkube-node-7cgxg\" (UID: \"95709b19-3c85-4ef1-9110-c23a97847732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638187 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-sysctl-d\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638200 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-system-cni-dir\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638130 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-socket-dir-parent\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638214 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72993a50-bf5d-40c5-833b-afff9f86e61d-crio-socket\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638239 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-lib-modules\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638242 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72993a50-bf5d-40c5-833b-afff9f86e61d-crio-socket\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638263 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-run\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638294 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-sysctl-d\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638301 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-run\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638294 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-sys-fs\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638324 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-lib-modules\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638338 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-sys-fs\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638341 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-tls\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638366 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-var-lib-cni-multus\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638381 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-conf-dir\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638403 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9933e90-d394-4e89-8e0f-115298c359fa-host-slash\") pod \"iptables-alerter-db4lm\" (UID: \"e9933e90-d394-4e89-8e0f-115298c359fa\") " pod="openshift-network-operator/iptables-alerter-db4lm" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638428 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-cnibin\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638430 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-host-var-lib-cni-multus\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638435 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-multus-conf-dir\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638450 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-var-lib-kubelet\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638475 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-kubernetes\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638479 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9933e90-d394-4e89-8e0f-115298c359fa-host-slash\") pod \"iptables-alerter-db4lm\" (UID: \"e9933e90-d394-4e89-8e0f-115298c359fa\") " pod="openshift-network-operator/iptables-alerter-db4lm" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638484 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-cnibin\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638525 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-registration-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638539 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-etc-kubernetes\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638553 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd6cf358-80f3-42f6-9848-6386a937a57a-cnibin\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638553 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/038ae7b2-90e2-44e6-9437-fcb4a0bcfda0-var-lib-kubelet\") pod \"tuned-khz68\" (UID: \"038ae7b2-90e2-44e6-9437-fcb4a0bcfda0\") " pod="openshift-cluster-node-tuning-operator/tuned-khz68" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638573 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7af86f87-5d83-4a8b-a0cc-88d4641406d2-registration-dir\") pod \"aws-ebs-csi-driver-node-gdgj7\" (UID: \"7af86f87-5d83-4a8b-a0cc-88d4641406d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdgj7" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638577 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-etc-kubernetes\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638612 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd6cf358-80f3-42f6-9848-6386a937a57a-cnibin\") pod \"multus-additional-cni-plugins-v5l7g\" (UID: \"fd6cf358-80f3-42f6-9848-6386a937a57a\") " pod="openshift-multus/multus-additional-cni-plugins-v5l7g" Oct 30 17:52:20.638968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.638636 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/471ac5c3-1cdd-4d79-bdc1-3acce27cc042-etc-kubernetes\") pod \"multus-th8wr\" (UID: \"471ac5c3-1cdd-4d79-bdc1-3acce27cc042\") " pod="openshift-multus/multus-th8wr" Oct 30 17:52:20.641319 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.639868 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-tls\") pod \"image-registry-69885b9d-n2bmz\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:20.641319 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.640533 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkgjk\" (UID: \"f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkgjk" Oct 30 17:52:20.641319 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.640590 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-tls\") pod \"image-registry-64bf8b8bb7-dlchr\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:20.702013 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.701982 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:20.705202 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.705183 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-kkgjk" Oct 30 17:52:20.709584 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.709567 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:20.967098 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:20.967031 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:21.058028 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:21.057919 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64bf8b8bb7-dlchr"] Oct 30 17:52:21.062153 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:21.062117 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a54420_0e75_4d4b_93cd_fe72c0cc2070.slice/crio-eb21e10fa52811064d3d3f5284e47863b7c0cb4586254f907f13d21de755213d WatchSource:0}: Error finding container eb21e10fa52811064d3d3f5284e47863b7c0cb4586254f907f13d21de755213d: Status 404 returned error can't find the container with id eb21e10fa52811064d3d3f5284e47863b7c0cb4586254f907f13d21de755213d Oct 30 17:52:21.071381 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:21.071354 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-kkgjk"] Oct 30 17:52:21.075324 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:21.075301 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf17fb4c5_0014_4bef_bd0a_7a4f3fc09cf2.slice/crio-b997dca3d841e7b8791de9d6ae55f435479d671f9c3723baecd45fede2bb6b99 WatchSource:0}: Error finding container b997dca3d841e7b8791de9d6ae55f435479d671f9c3723baecd45fede2bb6b99: Status 404 returned error can't find the container with id b997dca3d841e7b8791de9d6ae55f435479d671f9c3723baecd45fede2bb6b99 Oct 30 17:52:21.077216 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:21.077198 4603 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:52:21.103020 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:21.102999 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69885b9d-n2bmz"] Oct 30 17:52:21.105558 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:21.105536 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b06cbb4_e7da_4fc8_b68f_a54ce08d8c63.slice/crio-fad55bc523031a8ff5a57865db5276e7aa2e319a089fb5680d5a99dc37ce8a6a WatchSource:0}: Error finding container fad55bc523031a8ff5a57865db5276e7aa2e319a089fb5680d5a99dc37ce8a6a: Status 404 returned error can't find the container with id fad55bc523031a8ff5a57865db5276e7aa2e319a089fb5680d5a99dc37ce8a6a Oct 30 17:52:21.143934 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:21.143907 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:21.144034 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:21.143945 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:21.144034 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:21.143970 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:21.144034 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:21.143991 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs\") pod \"network-metrics-daemon-jnxk7\" (UID: \"827bbba0-b306-43db-b805-8c87dfabb874\") " pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:52:21.144131 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:21.144098 4603 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:52:21.144131 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:21.144109 4603 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:52:21.144131 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:21.144127 4603 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:52:21.144229 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:21.144129 4603 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:52:21.144229 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:21.144154 4603 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs podName:827bbba0-b306-43db-b805-8c87dfabb874 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:22.144137176 +0000 UTC m=+2.859707048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs") pod "network-metrics-daemon-jnxk7" (UID: "827bbba0-b306-43db-b805-8c87dfabb874") : secret "metrics-daemon-secret" not found Oct 30 17:52:21.144229 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:21.144171 4603 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-rbac-proxy-cm podName:72993a50-bf5d-40c5-833b-afff9f86e61d nodeName:}" failed. No retries permitted until 2026-10-30 17:52:22.144162512 +0000 UTC m=+2.859732365 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nsw4r" (UID: "72993a50-bf5d-40c5-833b-afff9f86e61d") : configmap "kube-rbac-proxy" not found Oct 30 17:52:21.144229 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:21.144182 4603 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls podName:72993a50-bf5d-40c5-833b-afff9f86e61d nodeName:}" failed. No retries permitted until 2026-10-30 17:52:22.144176842 +0000 UTC m=+2.859746694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nsw4r" (UID: "72993a50-bf5d-40c5-833b-afff9f86e61d") : secret "insights-runtime-extractor-tls" not found Oct 30 17:52:21.144229 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:21.144191 4603 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls podName:8e818352-df23-46b4-84d9-81fd259200ec nodeName:}" failed. No retries permitted until 2026-10-30 17:52:22.144186041 +0000 UTC m=+2.859755893 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls") pod "dns-default-7r5h8" (UID: "8e818352-df23-46b4-84d9-81fd259200ec") : secret "dns-default-metrics-tls" not found Oct 30 17:52:21.258557 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:21.258475 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:21.370624 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:21.370581 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7txkr_c9602b28-165a-4c17-96fa-9f0e25fd2cf8/serve-healthcheck-canary/0.log" Oct 30 17:52:21.452419 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:21.452383 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69885b9d-n2bmz" event={"ID":"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63","Type":"ContainerStarted","Data":"014a57414abff45c1ab258b1464c6416721556ed4d0d44d653df0e7844ae70a8"} Oct 30 17:52:21.452612 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:21.452445 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69885b9d-n2bmz" event={"ID":"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63","Type":"ContainerStarted","Data":"fad55bc523031a8ff5a57865db5276e7aa2e319a089fb5680d5a99dc37ce8a6a"} Oct 30 17:52:21.454116 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:21.454092 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" event={"ID":"21a54420-0e75-4d4b-93cd-fe72c0cc2070","Type":"ContainerStarted","Data":"d467cdc280024a088c2b415abd8fd30ddf377c5eee40bdc203791a9e024a790a"} Oct 30 17:52:21.454232 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:21.454136 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" event={"ID":"21a54420-0e75-4d4b-93cd-fe72c0cc2070","Type":"ContainerStarted","Data":"eb21e10fa52811064d3d3f5284e47863b7c0cb4586254f907f13d21de755213d"} Oct 30 17:52:21.455394 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:21.455369 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-kkgjk" event={"ID":"f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2","Type":"ContainerStarted","Data":"b997dca3d841e7b8791de9d6ae55f435479d671f9c3723baecd45fede2bb6b99"} Oct 30 17:52:21.765767 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:21.765730 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:21.991922 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:21.991881 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-jr2r5_1a6fe435-1778-45b0-ae18-6695f5824229/migrator/0.log" Oct 30 17:52:22.145841 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:22.145797 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:22.152778 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:22.152757 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:22.152837 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:22.152789 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:22.152837 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:22.152819 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:22.152916 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:22.152847 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs\") pod \"network-metrics-daemon-jnxk7\" (UID: \"827bbba0-b306-43db-b805-8c87dfabb874\") " pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:52:22.152916 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:22.152907 4603 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:52:22.152982 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:22.152969 4603 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:52:22.153026 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:22.152909 4603 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:52:22.153071 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:22.153032 4603 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls podName:8e818352-df23-46b4-84d9-81fd259200ec nodeName:}" failed. No retries permitted until 2026-10-30 17:52:24.153018051 +0000 UTC m=+4.868587917 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls") pod "dns-default-7r5h8" (UID: "8e818352-df23-46b4-84d9-81fd259200ec") : secret "dns-default-metrics-tls" not found Oct 30 17:52:22.153071 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:22.153047 4603 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs podName:827bbba0-b306-43db-b805-8c87dfabb874 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:24.153041315 +0000 UTC m=+4.868611167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs") pod "network-metrics-daemon-jnxk7" (UID: "827bbba0-b306-43db-b805-8c87dfabb874") : secret "metrics-daemon-secret" not found Oct 30 17:52:22.153156 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:22.153076 4603 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls podName:72993a50-bf5d-40c5-833b-afff9f86e61d nodeName:}" failed. No retries permitted until 2026-10-30 17:52:24.153052787 +0000 UTC m=+4.868622656 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nsw4r" (UID: "72993a50-bf5d-40c5-833b-afff9f86e61d") : secret "insights-runtime-extractor-tls" not found Oct 30 17:52:22.153334 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:22.153316 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72993a50-bf5d-40c5-833b-afff9f86e61d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:22.285765 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:22.285733 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:22.370888 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:22.370867 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf_b4d17d89-b883-472d-84c9-b8f4d0c8eb7e/kube-storage-version-migrator-operator/0.log" Oct 30 17:52:22.425680 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:22.425620 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:22.458537 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:22.458515 4603 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:52:22.458608 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:22.458521 4603 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:52:22.568113 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:22.568090 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:22.960983 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:22.960954 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:23.462890 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:23.462861 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-kkgjk" event={"ID":"f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2","Type":"ContainerStarted","Data":"cb0bcf0d7238229647fa6d756074c5f0a30ce35797be0b8befe5c4d424c8c8a7"} Oct 30 17:52:23.766294 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:23.766214 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:24.168431 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:24.168393 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:24.168663 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:24.168433 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:24.168663 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:24.168481 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs\") pod \"network-metrics-daemon-jnxk7\" (UID: \"827bbba0-b306-43db-b805-8c87dfabb874\") " pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:52:24.168663 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:24.168570 4603 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:52:24.168663 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:24.168646 4603 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls podName:72993a50-bf5d-40c5-833b-afff9f86e61d nodeName:}" failed. No retries permitted until 2026-10-30 17:52:28.168626445 +0000 UTC m=+8.884196314 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nsw4r" (UID: "72993a50-bf5d-40c5-833b-afff9f86e61d") : secret "insights-runtime-extractor-tls" not found Oct 30 17:52:24.168663 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:24.168642 4603 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:52:24.168937 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:24.168714 4603 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs podName:827bbba0-b306-43db-b805-8c87dfabb874 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:28.168695368 +0000 UTC m=+8.884265239 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs") pod "network-metrics-daemon-jnxk7" (UID: "827bbba0-b306-43db-b805-8c87dfabb874") : secret "metrics-daemon-secret" not found Oct 30 17:52:24.168937 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:24.168869 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:24.171435 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:24.171405 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e818352-df23-46b4-84d9-81fd259200ec-metrics-tls\") pod \"dns-default-7r5h8\" (UID: \"8e818352-df23-46b4-84d9-81fd259200ec\") " pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:24.310073 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:24.310043 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:24.363885 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:24.363677 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:24.568812 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:24.568781 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:24.619750 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:24.619725 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7r5h8"] Oct 30 17:52:24.623617 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:24.623589 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e818352_df23_46b4_84d9_81fd259200ec.slice/crio-c4e4575a75d7c25754a90572cee24706a680c5069e95a22d740863bedc7b9489 WatchSource:0}: Error finding container c4e4575a75d7c25754a90572cee24706a680c5069e95a22d740863bedc7b9489: Status 404 returned error can't find the container with id c4e4575a75d7c25754a90572cee24706a680c5069e95a22d740863bedc7b9489 Oct 30 17:52:24.814932 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:24.814894 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:24.957235 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:24.957177 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:25.110756 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:25.110721 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:25.258821 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:25.258736 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:25.402016 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:25.401968 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:25.471022 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:25.470979 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7r5h8" event={"ID":"8e818352-df23-46b4-84d9-81fd259200ec","Type":"ContainerStarted","Data":"c4e4575a75d7c25754a90572cee24706a680c5069e95a22d740863bedc7b9489"} Oct 30 17:52:25.547659 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:25.547622 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:25.770793 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:25.770755 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:26.068998 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:26.068974 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:26.172652 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:26.172611 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:26.370494 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:26.370422 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:26.476220 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:26.476187 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7r5h8" event={"ID":"8e818352-df23-46b4-84d9-81fd259200ec","Type":"ContainerStarted","Data":"f264dc07701fdc4306fa123cbcb100f48e00a9828ade1a757e2230342581f6c2"} Oct 30 17:52:26.476368 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:26.476235 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7r5h8" event={"ID":"8e818352-df23-46b4-84d9-81fd259200ec","Type":"ContainerStarted","Data":"feed426e69d5c64c604f2b998b6731edd7b1745ef80b6dbbd3f9535e0b112d09"} Oct 30 17:52:26.476368 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:26.476315 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:26.564432 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:26.564407 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:26.967628 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:26.967600 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:27.793581 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:27.793543 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:27.975042 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:27.975019 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:28.200960 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:28.200888 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:28.200960 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:28.200929 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs\") pod \"network-metrics-daemon-jnxk7\" (UID: \"827bbba0-b306-43db-b805-8c87dfabb874\") " pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:52:28.203369 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:28.203336 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/827bbba0-b306-43db-b805-8c87dfabb874-metrics-certs\") pod \"network-metrics-daemon-jnxk7\" (UID: \"827bbba0-b306-43db-b805-8c87dfabb874\") " pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:52:28.203525 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:28.203391 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72993a50-bf5d-40c5-833b-afff9f86e61d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nsw4r\" (UID: \"72993a50-bf5d-40c5-833b-afff9f86e61d\") " pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:28.206272 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:28.206255 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnxk7" Oct 30 17:52:28.210824 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:28.210801 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nsw4r" Oct 30 17:52:28.211704 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:28.211679 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:28.285493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:28.285228 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:28.364411 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:28.364383 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:28.483227 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:28.483168 4603 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:52:28.515491 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:28.515448 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nsw4r"] Oct 30 17:52:28.518712 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:28.518689 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72993a50_bf5d_40c5_833b_afff9f86e61d.slice/crio-7f4a94eb9fe7b40914ba412a298accab047a5db92e7d06bc23cd6368b52bc811 WatchSource:0}: Error finding container 7f4a94eb9fe7b40914ba412a298accab047a5db92e7d06bc23cd6368b52bc811: Status 404 returned error can't find the container with id 7f4a94eb9fe7b40914ba412a298accab047a5db92e7d06bc23cd6368b52bc811 Oct 30 17:52:28.527466 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:28.527447 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jnxk7"] Oct 30 17:52:28.530683 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:28.530657 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod827bbba0_b306_43db_b805_8c87dfabb874.slice/crio-60d2f8ec2cf986fd828d3a541d394e6ba09e275ac98ea50c2390a580549462c1 WatchSource:0}: Error finding container 60d2f8ec2cf986fd828d3a541d394e6ba09e275ac98ea50c2390a580549462c1: Status 404 returned error can't find the container with id 60d2f8ec2cf986fd828d3a541d394e6ba09e275ac98ea50c2390a580549462c1 Oct 30 17:52:28.571862 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:28.571836 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:28.954737 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:28.954661 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:29.166324 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:29.166297 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:29.489008 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:29.488980 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nsw4r" event={"ID":"72993a50-bf5d-40c5-833b-afff9f86e61d","Type":"ContainerStarted","Data":"8e43c317be2c74f8717f4d03025024e6a59ec2a7ac30bd446f66bed46ad5b102"} Oct 30 17:52:29.489135 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:29.489051 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nsw4r" event={"ID":"72993a50-bf5d-40c5-833b-afff9f86e61d","Type":"ContainerStarted","Data":"0ceb668bd41688d0e841faa872826db2530fc89be42df50d4befc0a406d72de0"} Oct 30 17:52:29.489135 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:29.489084 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nsw4r" event={"ID":"72993a50-bf5d-40c5-833b-afff9f86e61d","Type":"ContainerStarted","Data":"7f4a94eb9fe7b40914ba412a298accab047a5db92e7d06bc23cd6368b52bc811"} Oct 30 17:52:29.490903 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:29.490874 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jnxk7" event={"ID":"827bbba0-b306-43db-b805-8c87dfabb874","Type":"ContainerStarted","Data":"ca55046c610f4d3b17d27b37da9d3cd1056317232914517db26d3f00a25e3c31"} Oct 30 17:52:29.491012 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:29.490929 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jnxk7" event={"ID":"827bbba0-b306-43db-b805-8c87dfabb874","Type":"ContainerStarted","Data":"60d2f8ec2cf986fd828d3a541d394e6ba09e275ac98ea50c2390a580549462c1"} Oct 30 17:52:29.813401 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:29.813373 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:52:29.921178 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:29.921145 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:30.318606 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:30.318578 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:30.495534 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:30.495485 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nsw4r" event={"ID":"72993a50-bf5d-40c5-833b-afff9f86e61d","Type":"ContainerStarted","Data":"cb19c15a8518b6f85eed079e1fcd70fead54b6f506b66cefdfb66e6eee9fa281"} Oct 30 17:52:30.497364 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:30.497332 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jnxk7" event={"ID":"827bbba0-b306-43db-b805-8c87dfabb874","Type":"ContainerStarted","Data":"ba745faac1a6d7fc2f750ad9fa8eac8099fdc4d9c66f14d765aa58a938776e34"} Oct 30 17:52:32.399825 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:32.399791 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pgghg" Oct 30 17:52:33.870167 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:33.870128 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:34.014072 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:34.014035 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:34.157337 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:34.157258 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:34.293827 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:34.293769 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:34.435957 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:34.435879 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:35.313623 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:35.313583 4603 patch_prober.go:29] interesting pod/image-registry-64bf8b8bb7-dlchr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 17:52:35.314034 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:35.313665 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" podUID="21a54420-0e75-4d4b-93cd-fe72c0cc2070" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:52:35.834745 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:35.834719 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7cgxg" Oct 30 17:52:36.074025 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:36.073987 4603 patch_prober.go:29] interesting pod/image-registry-64bf8b8bb7-dlchr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 17:52:36.074201 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:36.074057 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" podUID="21a54420-0e75-4d4b-93cd-fe72c0cc2070" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:52:36.279569 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:36.279525 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:36.428710 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:36.428681 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:36.481895 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:36.481872 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7r5h8" Oct 30 17:52:38.192255 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:38.192227 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64bf8b8bb7-dlchr"] Oct 30 17:52:38.196752 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:38.196722 4603 patch_prober.go:29] interesting pod/image-registry-64bf8b8bb7-dlchr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 17:52:38.196889 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:38.196782 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" podUID="21a54420-0e75-4d4b-93cd-fe72c0cc2070" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:52:38.365106 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:38.365075 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:38.520750 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:38.520716 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:38.675900 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:38.675869 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:40.323271 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:40.323239 4603 patch_prober.go:29] interesting pod/image-registry-69885b9d-n2bmz container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 17:52:40.323651 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:40.323293 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-69885b9d-n2bmz" podUID="9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:52:40.500546 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:40.500494 4603 patch_prober.go:29] interesting pod/image-registry-69885b9d-n2bmz container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 17:52:40.500694 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:40.500555 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-69885b9d-n2bmz" podUID="9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:52:40.515880 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:40.515845 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:40.656996 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:40.656919 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:41.277999 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:41.277958 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:41.422260 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:41.422229 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:41.594565 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:41.594465 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:44.215748 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:44.215712 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:45.704697 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:45.704657 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:45.863285 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:45.863253 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:46.001671 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.001605 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5rlkk"] Oct 30 17:52:46.022994 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.022972 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.052237 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.052217 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-58pv5\"" Oct 30 17:52:46.052352 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.052247 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 17:52:46.052352 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.052270 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 17:52:46.056938 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.056914 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:46.085929 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.085898 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 17:52:46.085929 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.085913 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 17:52:46.086076 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.085929 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 17:52:46.086076 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.085937 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 17:52:46.114321 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.114299 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbbdea07-4cce-465f-a085-3cd5a0e5e334-metrics-client-ca\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.114416 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.114327 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bbbdea07-4cce-465f-a085-3cd5a0e5e334-root\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.114416 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.114343 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbbdea07-4cce-465f-a085-3cd5a0e5e334-sys\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.114416 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.114360 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bbbdea07-4cce-465f-a085-3cd5a0e5e334-node-exporter-textfile\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.114568 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.114460 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bbbdea07-4cce-465f-a085-3cd5a0e5e334-node-exporter-tls\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.114568 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.114485 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m24wt\" (UniqueName: \"kubernetes.io/projected/bbbdea07-4cce-465f-a085-3cd5a0e5e334-kube-api-access-m24wt\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.114568 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.114541 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbbdea07-4cce-465f-a085-3cd5a0e5e334-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.114693 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.114613 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bbbdea07-4cce-465f-a085-3cd5a0e5e334-node-exporter-wtmp\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.114693 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.114654 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bbbdea07-4cce-465f-a085-3cd5a0e5e334-node-exporter-accelerators-collector-config\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.215235 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.215210 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bbbdea07-4cce-465f-a085-3cd5a0e5e334-node-exporter-textfile\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.215339 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.215257 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bbbdea07-4cce-465f-a085-3cd5a0e5e334-node-exporter-tls\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.215339 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.215280 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m24wt\" (UniqueName: \"kubernetes.io/projected/bbbdea07-4cce-465f-a085-3cd5a0e5e334-kube-api-access-m24wt\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.215339 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.215306 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbbdea07-4cce-465f-a085-3cd5a0e5e334-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.215339 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.215331 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bbbdea07-4cce-465f-a085-3cd5a0e5e334-node-exporter-wtmp\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.215570 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.215351 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bbbdea07-4cce-465f-a085-3cd5a0e5e334-node-exporter-accelerators-collector-config\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.215570 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.215442 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbbdea07-4cce-465f-a085-3cd5a0e5e334-metrics-client-ca\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.215570 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.215486 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bbbdea07-4cce-465f-a085-3cd5a0e5e334-root\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.215570 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.215540 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbbdea07-4cce-465f-a085-3cd5a0e5e334-sys\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.215766 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.215542 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bbbdea07-4cce-465f-a085-3cd5a0e5e334-node-exporter-wtmp\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.215766 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.215581 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bbbdea07-4cce-465f-a085-3cd5a0e5e334-root\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.215766 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.215594 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bbbdea07-4cce-465f-a085-3cd5a0e5e334-node-exporter-textfile\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.215766 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.215623 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbbdea07-4cce-465f-a085-3cd5a0e5e334-sys\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.215994 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.215973 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bbbdea07-4cce-465f-a085-3cd5a0e5e334-node-exporter-accelerators-collector-config\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.216177 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.216154 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbbdea07-4cce-465f-a085-3cd5a0e5e334-metrics-client-ca\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.217854 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.217824 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bbbdea07-4cce-465f-a085-3cd5a0e5e334-node-exporter-tls\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.217967 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.217914 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbbdea07-4cce-465f-a085-3cd5a0e5e334-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.218386 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.218361 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:46.239348 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.239329 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m24wt\" (UniqueName: \"kubernetes.io/projected/bbbdea07-4cce-465f-a085-3cd5a0e5e334-kube-api-access-m24wt\") pod \"node-exporter-5rlkk\" (UID: \"bbbdea07-4cce-465f-a085-3cd5a0e5e334\") " pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.368144 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.368115 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5rlkk" Oct 30 17:52:46.398426 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:46.398401 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbdea07_4cce_465f_a085_3cd5a0e5e334.slice/crio-de9f977e46c7d894229afa93a912e0ff2dd16e6ffe327e90533f247015fa0443 WatchSource:0}: Error finding container de9f977e46c7d894229afa93a912e0ff2dd16e6ffe327e90533f247015fa0443: Status 404 returned error can't find the container with id de9f977e46c7d894229afa93a912e0ff2dd16e6ffe327e90533f247015fa0443 Oct 30 17:52:46.399862 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.399839 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:46.542993 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.542965 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:46.548285 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.548265 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5rlkk" event={"ID":"bbbdea07-4cce-465f-a085-3cd5a0e5e334","Type":"ContainerStarted","Data":"de9f977e46c7d894229afa93a912e0ff2dd16e6ffe327e90533f247015fa0443"} Oct 30 17:52:46.729624 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:46.729556 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:47.144097 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:47.144065 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:47.297017 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:47.296979 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:47.513317 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:47.513236 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:48.196547 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:48.196464 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:52:48.556749 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:48.556718 4603 generic.go:356] "Generic (PLEG): container finished" podID="bbbdea07-4cce-465f-a085-3cd5a0e5e334" containerID="6bf334fbf791f35e40ca8d8d97cea8ec48471e92a80c3c464e875902c5fb7152" exitCode=0 Oct 30 17:52:48.556903 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:48.556799 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5rlkk" event={"ID":"bbbdea07-4cce-465f-a085-3cd5a0e5e334","Type":"ContainerDied","Data":"6bf334fbf791f35e40ca8d8d97cea8ec48471e92a80c3c464e875902c5fb7152"} Oct 30 17:52:49.226138 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:49.226101 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:49.401410 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:49.401378 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:49.544104 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:49.544076 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:49.561748 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:49.561723 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5rlkk" event={"ID":"bbbdea07-4cce-465f-a085-3cd5a0e5e334","Type":"ContainerStarted","Data":"fad1f9fbba7fe627362bb50a73591a89cc65a51e1d82f00ca208efc4aec18703"} Oct 30 17:52:49.561857 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:49.561773 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5rlkk" event={"ID":"bbbdea07-4cce-465f-a085-3cd5a0e5e334","Type":"ContainerStarted","Data":"a3a6324803106cd85ae48a6bf849ac750fecd4df2931c644f463ddefc03d70d9"} Oct 30 17:52:49.618144 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:49.618084 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5rlkk" podStartSLOduration=3.180967616 podStartE2EDuration="4.618069788s" totalImagesPullingTime="1.348262444s" totalInitContainerRuntime="88.839728ms" isStatefulPod=true podCreationTimestamp="2026-10-30 17:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:49.617460927 +0000 UTC m=+30.333030801" watchObservedRunningTime="2026-10-30 17:52:49.618069788 +0000 UTC m=+30.333639656" Oct 30 17:52:49.816450 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:49.816394 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4xxmr" Oct 30 17:52:50.275141 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.275099 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:50.323616 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.323589 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:52:50.417586 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.417559 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:50.511083 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.511055 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7dc5676d65-7f84v"] Oct 30 17:52:50.531253 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.531206 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:52:50.535679 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.535654 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7dc5676d65-7f84v"] Oct 30 17:52:50.545612 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.545584 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 17:52:50.545612 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.545605 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 17:52:50.545743 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.545712 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5jn20iiqgbmh6\"" Oct 30 17:52:50.545743 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.545719 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 17:52:50.545806 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.545741 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-tx9q6\"" Oct 30 17:52:50.545806 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.545784 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 17:52:50.650568 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.650539 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0a90475d-0a43-43eb-9aeb-42804d607d3b-client-ca-bundle\") pod \"metrics-server-7dc5676d65-7f84v\" (UID: \"0a90475d-0a43-43eb-9aeb-42804d607d3b\") " pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:52:50.650682 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.650581 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdqg4\" (UniqueName: \"kubernetes.io/projected/0a90475d-0a43-43eb-9aeb-42804d607d3b-kube-api-access-sdqg4\") pod \"metrics-server-7dc5676d65-7f84v\" (UID: \"0a90475d-0a43-43eb-9aeb-42804d607d3b\") " pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:52:50.650682 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.650606 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0a90475d-0a43-43eb-9aeb-42804d607d3b-secret-metrics-server-tls\") pod \"metrics-server-7dc5676d65-7f84v\" (UID: \"0a90475d-0a43-43eb-9aeb-42804d607d3b\") " pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:52:50.650775 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.650703 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0a90475d-0a43-43eb-9aeb-42804d607d3b-metrics-server-audit-profiles\") pod \"metrics-server-7dc5676d65-7f84v\" (UID: \"0a90475d-0a43-43eb-9aeb-42804d607d3b\") " pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:52:50.650810 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.650796 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a90475d-0a43-43eb-9aeb-42804d607d3b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7dc5676d65-7f84v\" (UID: \"0a90475d-0a43-43eb-9aeb-42804d607d3b\") " pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:52:50.650842 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.650821 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0a90475d-0a43-43eb-9aeb-42804d607d3b-audit-log\") pod \"metrics-server-7dc5676d65-7f84v\" (UID: \"0a90475d-0a43-43eb-9aeb-42804d607d3b\") " pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:52:50.650884 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.650860 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0a90475d-0a43-43eb-9aeb-42804d607d3b-secret-metrics-server-client-certs\") pod \"metrics-server-7dc5676d65-7f84v\" (UID: \"0a90475d-0a43-43eb-9aeb-42804d607d3b\") " pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:52:50.752058 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.752033 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0a90475d-0a43-43eb-9aeb-42804d607d3b-client-ca-bundle\") pod \"metrics-server-7dc5676d65-7f84v\" (UID: \"0a90475d-0a43-43eb-9aeb-42804d607d3b\") " pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:52:50.752205 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.752089 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdqg4\" (UniqueName: \"kubernetes.io/projected/0a90475d-0a43-43eb-9aeb-42804d607d3b-kube-api-access-sdqg4\") pod \"metrics-server-7dc5676d65-7f84v\" (UID: \"0a90475d-0a43-43eb-9aeb-42804d607d3b\") " pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:52:50.752205 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.752130 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0a90475d-0a43-43eb-9aeb-42804d607d3b-secret-metrics-server-tls\") pod \"metrics-server-7dc5676d65-7f84v\" (UID: \"0a90475d-0a43-43eb-9aeb-42804d607d3b\") " pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:52:50.752370 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.752347 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0a90475d-0a43-43eb-9aeb-42804d607d3b-metrics-server-audit-profiles\") pod \"metrics-server-7dc5676d65-7f84v\" (UID: \"0a90475d-0a43-43eb-9aeb-42804d607d3b\") " pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:52:50.752438 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.752393 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a90475d-0a43-43eb-9aeb-42804d607d3b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7dc5676d65-7f84v\" (UID: \"0a90475d-0a43-43eb-9aeb-42804d607d3b\") " pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:52:50.752438 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.752432 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0a90475d-0a43-43eb-9aeb-42804d607d3b-audit-log\") pod \"metrics-server-7dc5676d65-7f84v\" (UID: \"0a90475d-0a43-43eb-9aeb-42804d607d3b\") " pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:52:50.752568 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.752492 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0a90475d-0a43-43eb-9aeb-42804d607d3b-secret-metrics-server-client-certs\") pod \"metrics-server-7dc5676d65-7f84v\" (UID: \"0a90475d-0a43-43eb-9aeb-42804d607d3b\") " pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:52:50.752894 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.752849 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0a90475d-0a43-43eb-9aeb-42804d607d3b-audit-log\") pod \"metrics-server-7dc5676d65-7f84v\" (UID: \"0a90475d-0a43-43eb-9aeb-42804d607d3b\") " pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:52:50.753118 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.753101 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a90475d-0a43-43eb-9aeb-42804d607d3b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7dc5676d65-7f84v\" (UID: \"0a90475d-0a43-43eb-9aeb-42804d607d3b\") " pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:52:50.753614 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.753593 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0a90475d-0a43-43eb-9aeb-42804d607d3b-metrics-server-audit-profiles\") pod \"metrics-server-7dc5676d65-7f84v\" (UID: \"0a90475d-0a43-43eb-9aeb-42804d607d3b\") " pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:52:50.754850 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.754825 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0a90475d-0a43-43eb-9aeb-42804d607d3b-client-ca-bundle\") pod \"metrics-server-7dc5676d65-7f84v\" (UID: \"0a90475d-0a43-43eb-9aeb-42804d607d3b\") " pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:52:50.755056 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.755037 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0a90475d-0a43-43eb-9aeb-42804d607d3b-secret-metrics-server-client-certs\") pod \"metrics-server-7dc5676d65-7f84v\" (UID: \"0a90475d-0a43-43eb-9aeb-42804d607d3b\") " pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:52:50.755211 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.755187 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0a90475d-0a43-43eb-9aeb-42804d607d3b-secret-metrics-server-tls\") pod \"metrics-server-7dc5676d65-7f84v\" (UID: \"0a90475d-0a43-43eb-9aeb-42804d607d3b\") " pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:52:50.775258 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.775229 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdqg4\" (UniqueName: \"kubernetes.io/projected/0a90475d-0a43-43eb-9aeb-42804d607d3b-kube-api-access-sdqg4\") pod \"metrics-server-7dc5676d65-7f84v\" (UID: \"0a90475d-0a43-43eb-9aeb-42804d607d3b\") " pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:52:50.781447 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.781397 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:50.858067 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.858044 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:52:50.931346 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:50.931319 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:51.143150 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.143081 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:51.169652 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.169630 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7dc5676d65-7f84v"] Oct 30 17:52:51.172423 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:51.172388 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a90475d_0a43_43eb_9aeb_42804d607d3b.slice/crio-79b706c7c6b16d49e7134d33d6b46bdcaf5013bdcd9f7d6b3f97f46881f3a8d5 WatchSource:0}: Error finding container 79b706c7c6b16d49e7134d33d6b46bdcaf5013bdcd9f7d6b3f97f46881f3a8d5: Status 404 returned error can't find the container with id 79b706c7c6b16d49e7134d33d6b46bdcaf5013bdcd9f7d6b3f97f46881f3a8d5 Oct 30 17:52:51.326609 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.326584 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-76db67c9b-cvzv8"] Oct 30 17:52:51.329375 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.329360 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.344555 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.344534 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 17:52:51.344959 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.344942 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 17:52:51.345044 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.344962 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 17:52:51.345044 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.344972 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 17:52:51.345044 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.344963 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 17:52:51.345044 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.345025 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2d2zq\"" Oct 30 17:52:51.368420 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.368399 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 17:52:51.369251 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.369228 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76db67c9b-cvzv8"] Oct 30 17:52:51.456818 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.456766 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b970fffa-a462-4914-bec0-3220c1b90dae-secret-telemeter-client\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.456818 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.456795 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b970fffa-a462-4914-bec0-3220c1b90dae-federate-client-tls\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.456974 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.456829 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b970fffa-a462-4914-bec0-3220c1b90dae-telemeter-client-tls\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.456974 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.456955 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b970fffa-a462-4914-bec0-3220c1b90dae-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.457064 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.456982 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b970fffa-a462-4914-bec0-3220c1b90dae-metrics-client-ca\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.457064 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.457008 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b970fffa-a462-4914-bec0-3220c1b90dae-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.457064 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.457024 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dd97\" (UniqueName: \"kubernetes.io/projected/b970fffa-a462-4914-bec0-3220c1b90dae-kube-api-access-5dd97\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.457175 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.457139 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b970fffa-a462-4914-bec0-3220c1b90dae-serving-certs-ca-bundle\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.490088 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.490060 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:51.557890 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.557861 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b970fffa-a462-4914-bec0-3220c1b90dae-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.558074 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.557898 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dd97\" (UniqueName: \"kubernetes.io/projected/b970fffa-a462-4914-bec0-3220c1b90dae-kube-api-access-5dd97\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.558074 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.557945 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b970fffa-a462-4914-bec0-3220c1b90dae-serving-certs-ca-bundle\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.558074 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.557986 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b970fffa-a462-4914-bec0-3220c1b90dae-secret-telemeter-client\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.558074 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.558020 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b970fffa-a462-4914-bec0-3220c1b90dae-federate-client-tls\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.558074 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.558066 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b970fffa-a462-4914-bec0-3220c1b90dae-telemeter-client-tls\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.558329 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.558107 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b970fffa-a462-4914-bec0-3220c1b90dae-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.558329 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.558136 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b970fffa-a462-4914-bec0-3220c1b90dae-metrics-client-ca\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.558881 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.558858 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b970fffa-a462-4914-bec0-3220c1b90dae-metrics-client-ca\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.558999 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.558922 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b970fffa-a462-4914-bec0-3220c1b90dae-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.559461 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.559417 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b970fffa-a462-4914-bec0-3220c1b90dae-serving-certs-ca-bundle\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.561403 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.561385 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b970fffa-a462-4914-bec0-3220c1b90dae-federate-client-tls\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.561802 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.561781 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b970fffa-a462-4914-bec0-3220c1b90dae-secret-telemeter-client\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.561928 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.561906 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b970fffa-a462-4914-bec0-3220c1b90dae-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.562053 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.561996 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b970fffa-a462-4914-bec0-3220c1b90dae-telemeter-client-tls\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.569445 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.569414 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" event={"ID":"0a90475d-0a43-43eb-9aeb-42804d607d3b","Type":"ContainerStarted","Data":"79b706c7c6b16d49e7134d33d6b46bdcaf5013bdcd9f7d6b3f97f46881f3a8d5"} Oct 30 17:52:51.580975 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.580954 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dd97\" (UniqueName: \"kubernetes.io/projected/b970fffa-a462-4914-bec0-3220c1b90dae-kube-api-access-5dd97\") pod \"telemeter-client-76db67c9b-cvzv8\" (UID: \"b970fffa-a462-4914-bec0-3220c1b90dae\") " pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.636165 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.636138 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:51.641050 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.640986 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" Oct 30 17:52:51.932785 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:52:51.932634 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 17:52:51.981807 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:51.981780 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76db67c9b-cvzv8"] Oct 30 17:52:51.998276 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:52:51.998247 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb970fffa_a462_4914_bec0_3220c1b90dae.slice/crio-ac5d49193c429a42b033d3e6122fb8c8fe010e304f7746c9a20c6d40807d7bde WatchSource:0}: Error finding container ac5d49193c429a42b033d3e6122fb8c8fe010e304f7746c9a20c6d40807d7bde: Status 404 returned error can't find the container with id ac5d49193c429a42b033d3e6122fb8c8fe010e304f7746c9a20c6d40807d7bde Oct 30 17:52:52.575619 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:52.575532 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" event={"ID":"0a90475d-0a43-43eb-9aeb-42804d607d3b","Type":"ContainerStarted","Data":"53eb2c0df7dd81dfe922ab3609be8c015a74cad7b933e19b7762f09f2196937b"} Oct 30 17:52:52.577102 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:52.577061 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" event={"ID":"b970fffa-a462-4914-bec0-3220c1b90dae","Type":"ContainerStarted","Data":"ac5d49193c429a42b033d3e6122fb8c8fe010e304f7746c9a20c6d40807d7bde"} Oct 30 17:52:52.617944 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:52.617899 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" podStartSLOduration=1.478811131 podStartE2EDuration="2.617885313s" totalImagesPullingTime="1.139074182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:52.617302613 +0000 UTC m=+33.332872512" watchObservedRunningTime="2026-10-30 17:52:52.617885313 +0000 UTC m=+33.333455186" Oct 30 17:52:53.115882 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:53.115846 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:54.585701 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:54.585670 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" event={"ID":"b970fffa-a462-4914-bec0-3220c1b90dae","Type":"ContainerStarted","Data":"029a7701ec63e03a1b4bcc239e5f79e6eab9dfba5f328410a9ca65365ab0baa5"} Oct 30 17:52:55.523831 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:55.523800 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:55.594805 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:55.594773 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" event={"ID":"b970fffa-a462-4914-bec0-3220c1b90dae","Type":"ContainerStarted","Data":"d08bfed7035c9ec2bf850b6e8dcb36bc3a39b7b5fb863314290e86d67fbd0e94"} Oct 30 17:52:55.595195 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:55.594853 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" event={"ID":"b970fffa-a462-4914-bec0-3220c1b90dae","Type":"ContainerStarted","Data":"a5d5d1ce5ca4da34fc7a470fea20b977029fe094da9071ab220c58ad4ee81013"} Oct 30 17:52:55.666679 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:55.666631 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-76db67c9b-cvzv8" podStartSLOduration=2.186343512 podStartE2EDuration="4.666617557s" totalImagesPullingTime="2.480274045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:55.66619823 +0000 UTC m=+36.381768117" watchObservedRunningTime="2026-10-30 17:52:55.666617557 +0000 UTC m=+36.382187444" Oct 30 17:52:55.681470 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:55.681446 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:57.896597 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:57.896558 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:58.104443 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:52:58.104413 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:00.234030 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:00.233983 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:00.377377 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:00.377345 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:01.838007 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:01.837977 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69885b9d-n2bmz"] Oct 30 17:53:01.975583 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:01.975549 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:02.156600 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:02.156521 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:02.499405 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:02.499312 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:02.672382 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:02.672359 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:03.213684 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.213648 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" podUID="21a54420-0e75-4d4b-93cd-fe72c0cc2070" containerName="registry" containerID="cri-o://d467cdc280024a088c2b415abd8fd30ddf377c5eee40bdc203791a9e024a790a" gracePeriod=30 Oct 30 17:53:03.456004 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.455982 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:53:03.535356 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.535332 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-tls\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-tls\") pod \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " Oct 30 17:53:03.535536 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.535394 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21a54420-0e75-4d4b-93cd-fe72c0cc2070-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21a54420-0e75-4d4b-93cd-fe72c0cc2070-image-registry-private-configuration\") pod \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " Oct 30 17:53:03.535536 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.535449 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21a54420-0e75-4d4b-93cd-fe72c0cc2070-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21a54420-0e75-4d4b-93cd-fe72c0cc2070-installation-pull-secrets\") pod \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " Oct 30 17:53:03.535536 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.535484 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-certificates\") pod \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " Oct 30 17:53:03.535705 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.535541 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21a54420-0e75-4d4b-93cd-fe72c0cc2070-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21a54420-0e75-4d4b-93cd-fe72c0cc2070-trusted-ca\") pod \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " Oct 30 17:53:03.535705 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.535567 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-bound-sa-token\") pod \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " Oct 30 17:53:03.535705 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.535619 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-kube-api-access-hmwhl\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-kube-api-access-hmwhl\") pod \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " Oct 30 17:53:03.535705 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.535663 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a54420-0e75-4d4b-93cd-fe72c0cc2070-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21a54420-0e75-4d4b-93cd-fe72c0cc2070-ca-trust-extracted\") pod \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\" (UID: \"21a54420-0e75-4d4b-93cd-fe72c0cc2070\") " Oct 30 17:53:03.536576 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.536288 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a54420-0e75-4d4b-93cd-fe72c0cc2070-trusted-ca" pod "21a54420-0e75-4d4b-93cd-fe72c0cc2070" (UID: "21a54420-0e75-4d4b-93cd-fe72c0cc2070"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:03.536576 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.536537 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-certificates" pod "21a54420-0e75-4d4b-93cd-fe72c0cc2070" (UID: "21a54420-0e75-4d4b-93cd-fe72c0cc2070"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:03.538340 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.538244 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-tls" pod "21a54420-0e75-4d4b-93cd-fe72c0cc2070" (UID: "21a54420-0e75-4d4b-93cd-fe72c0cc2070"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:53:03.538591 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.538565 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21a54420-0e75-4d4b-93cd-fe72c0cc2070-image-registry-private-configuration" pod "21a54420-0e75-4d4b-93cd-fe72c0cc2070" (UID: "21a54420-0e75-4d4b-93cd-fe72c0cc2070"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:03.538684 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.538661 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21a54420-0e75-4d4b-93cd-fe72c0cc2070-installation-pull-secrets" pod "21a54420-0e75-4d4b-93cd-fe72c0cc2070" (UID: "21a54420-0e75-4d4b-93cd-fe72c0cc2070"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:03.538905 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.538881 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-kube-api-access-hmwhl" pod "21a54420-0e75-4d4b-93cd-fe72c0cc2070" (UID: "21a54420-0e75-4d4b-93cd-fe72c0cc2070"). InnerVolumeSpecName "kube-api-access-hmwhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:53:03.539026 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.538966 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-bound-sa-token" pod "21a54420-0e75-4d4b-93cd-fe72c0cc2070" (UID: "21a54420-0e75-4d4b-93cd-fe72c0cc2070"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:53:03.543603 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.543577 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a54420-0e75-4d4b-93cd-fe72c0cc2070-ca-trust-extracted" pod "21a54420-0e75-4d4b-93cd-fe72c0cc2070" (UID: "21a54420-0e75-4d4b-93cd-fe72c0cc2070"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:03.619717 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.619693 4603 generic.go:356] "Generic (PLEG): container finished" podID="21a54420-0e75-4d4b-93cd-fe72c0cc2070" containerID="d467cdc280024a088c2b415abd8fd30ddf377c5eee40bdc203791a9e024a790a" exitCode=0 Oct 30 17:53:03.619806 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.619741 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" Oct 30 17:53:03.619806 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.619758 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" event={"ID":"21a54420-0e75-4d4b-93cd-fe72c0cc2070","Type":"ContainerDied","Data":"d467cdc280024a088c2b415abd8fd30ddf377c5eee40bdc203791a9e024a790a"} Oct 30 17:53:03.619806 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.619796 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bf8b8bb7-dlchr" event={"ID":"21a54420-0e75-4d4b-93cd-fe72c0cc2070","Type":"ContainerDied","Data":"eb21e10fa52811064d3d3f5284e47863b7c0cb4586254f907f13d21de755213d"} Oct 30 17:53:03.619900 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.619826 4603 scope.go:122] "RemoveContainer" containerID="d467cdc280024a088c2b415abd8fd30ddf377c5eee40bdc203791a9e024a790a" Oct 30 17:53:03.627903 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.627886 4603 scope.go:122] "RemoveContainer" containerID="d467cdc280024a088c2b415abd8fd30ddf377c5eee40bdc203791a9e024a790a" Oct 30 17:53:03.628189 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:53:03.628166 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d467cdc280024a088c2b415abd8fd30ddf377c5eee40bdc203791a9e024a790a\": container with ID starting with d467cdc280024a088c2b415abd8fd30ddf377c5eee40bdc203791a9e024a790a not found: ID does not exist" containerID="d467cdc280024a088c2b415abd8fd30ddf377c5eee40bdc203791a9e024a790a" Oct 30 17:53:03.628267 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.628203 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d467cdc280024a088c2b415abd8fd30ddf377c5eee40bdc203791a9e024a790a"} err="failed to get container status \"d467cdc280024a088c2b415abd8fd30ddf377c5eee40bdc203791a9e024a790a\": rpc error: code = NotFound desc = could not find container \"d467cdc280024a088c2b415abd8fd30ddf377c5eee40bdc203791a9e024a790a\": container with ID starting with d467cdc280024a088c2b415abd8fd30ddf377c5eee40bdc203791a9e024a790a not found: ID does not exist" Oct 30 17:53:03.636256 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.636232 4603 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-certificates\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:53:03.636256 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.636254 4603 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21a54420-0e75-4d4b-93cd-fe72c0cc2070-trusted-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:53:03.636390 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.636271 4603 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-bound-sa-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:53:03.636390 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.636285 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmwhl\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-kube-api-access-hmwhl\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:53:03.636390 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.636298 4603 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21a54420-0e75-4d4b-93cd-fe72c0cc2070-ca-trust-extracted\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:53:03.636390 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.636311 4603 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21a54420-0e75-4d4b-93cd-fe72c0cc2070-registry-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:53:03.636390 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.636326 4603 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21a54420-0e75-4d4b-93cd-fe72c0cc2070-image-registry-private-configuration\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:53:03.636390 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.636339 4603 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21a54420-0e75-4d4b-93cd-fe72c0cc2070-installation-pull-secrets\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:53:03.670418 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.670391 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64bf8b8bb7-dlchr"] Oct 30 17:53:03.680848 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:03.677765 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64bf8b8bb7-dlchr"] Oct 30 17:53:05.406864 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:05.406833 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21a54420-0e75-4d4b-93cd-fe72c0cc2070" path="/var/lib/kubelet/pods/21a54420-0e75-4d4b-93cd-fe72c0cc2070/volumes" Oct 30 17:53:07.711724 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:07.711689 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:07.860169 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:07.860138 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:10.858846 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:10.858814 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:53:10.859229 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:10.858872 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:53:11.335559 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:11.335497 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:11.505000 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:11.504971 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:21.927945 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:53:21.927905 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 17:53:25.513724 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:25.513683 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:25.663252 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:25.663212 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:26.857639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:26.857596 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-69885b9d-n2bmz" podUID="9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63" containerName="registry" containerID="cri-o://014a57414abff45c1ab258b1464c6416721556ed4d0d44d653df0e7844ae70a8" gracePeriod=30 Oct 30 17:53:27.099092 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.099070 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:53:27.190260 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.190197 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-installation-pull-secrets\") pod \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " Oct 30 17:53:27.190260 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.190244 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-bound-sa-token\") pod \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " Oct 30 17:53:27.190449 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.190279 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-certificates\") pod \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " Oct 30 17:53:27.190449 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.190314 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-image-registry-private-configuration\") pod \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " Oct 30 17:53:27.190449 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.190351 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-ca-trust-extracted\") pod \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " Oct 30 17:53:27.190449 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.190419 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-trusted-ca\") pod \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " Oct 30 17:53:27.190678 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.190455 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-kube-api-access-vgxhb\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-kube-api-access-vgxhb\") pod \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " Oct 30 17:53:27.190678 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.190527 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-tls\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-tls\") pod \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\" (UID: \"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63\") " Oct 30 17:53:27.190884 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.190851 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-certificates" pod "9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63" (UID: "9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:27.191368 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.191315 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-trusted-ca" pod "9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63" (UID: "9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:27.192798 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.192776 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-tls" pod "9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63" (UID: "9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:53:27.192904 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.192809 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-image-registry-private-configuration" pod "9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63" (UID: "9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:27.192968 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.192945 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-installation-pull-secrets" pod "9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63" (UID: "9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:27.193102 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.193085 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-bound-sa-token" pod "9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63" (UID: "9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:53:27.193404 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.193378 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-kube-api-access-vgxhb" pod "9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63" (UID: "9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63"). InnerVolumeSpecName "kube-api-access-vgxhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:53:27.198008 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.197981 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-ca-trust-extracted" pod "9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63" (UID: "9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:27.291530 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.291494 4603 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-image-registry-private-configuration\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:53:27.291530 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.291527 4603 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-ca-trust-extracted\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:53:27.291639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.291537 4603 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-trusted-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:53:27.291639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.291546 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgxhb\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-kube-api-access-vgxhb\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:53:27.291639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.291555 4603 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:53:27.291639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.291565 4603 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-installation-pull-secrets\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:53:27.291639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.291573 4603 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-bound-sa-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:53:27.291639 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.291584 4603 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63-registry-certificates\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:53:27.699853 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.699824 4603 generic.go:356] "Generic (PLEG): container finished" podID="9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63" containerID="014a57414abff45c1ab258b1464c6416721556ed4d0d44d653df0e7844ae70a8" exitCode=0 Oct 30 17:53:27.700003 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.699888 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69885b9d-n2bmz" event={"ID":"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63","Type":"ContainerDied","Data":"014a57414abff45c1ab258b1464c6416721556ed4d0d44d653df0e7844ae70a8"} Oct 30 17:53:27.700003 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.699894 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69885b9d-n2bmz" Oct 30 17:53:27.700003 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.699926 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69885b9d-n2bmz" event={"ID":"9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63","Type":"ContainerDied","Data":"fad55bc523031a8ff5a57865db5276e7aa2e319a089fb5680d5a99dc37ce8a6a"} Oct 30 17:53:27.700003 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.699944 4603 scope.go:122] "RemoveContainer" containerID="014a57414abff45c1ab258b1464c6416721556ed4d0d44d653df0e7844ae70a8" Oct 30 17:53:27.707659 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.707640 4603 scope.go:122] "RemoveContainer" containerID="014a57414abff45c1ab258b1464c6416721556ed4d0d44d653df0e7844ae70a8" Oct 30 17:53:27.707932 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:53:27.707909 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"014a57414abff45c1ab258b1464c6416721556ed4d0d44d653df0e7844ae70a8\": container with ID starting with 014a57414abff45c1ab258b1464c6416721556ed4d0d44d653df0e7844ae70a8 not found: ID does not exist" containerID="014a57414abff45c1ab258b1464c6416721556ed4d0d44d653df0e7844ae70a8" Oct 30 17:53:27.707984 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.707940 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"014a57414abff45c1ab258b1464c6416721556ed4d0d44d653df0e7844ae70a8"} err="failed to get container status \"014a57414abff45c1ab258b1464c6416721556ed4d0d44d653df0e7844ae70a8\": rpc error: code = NotFound desc = could not find container \"014a57414abff45c1ab258b1464c6416721556ed4d0d44d653df0e7844ae70a8\": container with ID starting with 014a57414abff45c1ab258b1464c6416721556ed4d0d44d653df0e7844ae70a8 not found: ID does not exist" Oct 30 17:53:27.760841 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.760821 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69885b9d-n2bmz"] Oct 30 17:53:27.765240 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:27.765224 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69885b9d-n2bmz"] Oct 30 17:53:29.408087 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:29.408054 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63" path="/var/lib/kubelet/pods/9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63/volumes" Oct 30 17:53:30.864065 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:30.864029 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:53:30.867902 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:30.867882 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7dc5676d65-7f84v" Oct 30 17:53:31.845392 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:31.845364 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:31.991670 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:31.991624 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:34.062565 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:34.062497 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:34.201863 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:34.201825 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:34.727696 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:34.727660 4603 generic.go:356] "Generic (PLEG): container finished" podID="b4d17d89-b883-472d-84c9-b8f4d0c8eb7e" containerID="3da433ccc715ed4d16a8024ef01191d057f3872bc6bb7dc9157724e6031ed464" exitCode=0 Oct 30 17:53:34.727853 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:34.727700 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf" event={"ID":"b4d17d89-b883-472d-84c9-b8f4d0c8eb7e","Type":"ContainerDied","Data":"3da433ccc715ed4d16a8024ef01191d057f3872bc6bb7dc9157724e6031ed464"} Oct 30 17:53:34.727983 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:34.727970 4603 scope.go:122] "RemoveContainer" containerID="3da433ccc715ed4d16a8024ef01191d057f3872bc6bb7dc9157724e6031ed464" Oct 30 17:53:35.733160 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:35.733127 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:35.733689 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:35.733381 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtnvf" event={"ID":"b4d17d89-b883-472d-84c9-b8f4d0c8eb7e","Type":"ContainerStarted","Data":"7ca3f50705fac1219dd583e282c94490030004fcc3b0949fd43671260c4ea8d6"} Oct 30 17:53:35.877288 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:35.877262 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:37.361465 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:37.361425 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:44.515342 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:44.515300 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:44.659149 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:44.659118 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:51.927742 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:53:51.927710 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 17:53:55.522903 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:55.522861 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:55.662519 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:53:55.662456 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:54:11.455904 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:54:11.455868 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:54:21.928465 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:54:21.928428 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 17:54:51.928466 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:54:51.928386 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 17:55:11.567622 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:55:11.567586 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:55:21.927797 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:55:21.927761 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 17:55:24.517843 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:55:24.517809 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:55:24.671561 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:55:24.671522 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:55:30.530075 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:55:30.530037 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:55:32.925849 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:55:32.925818 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:55:33.067374 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:55:33.067347 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:55:50.403974 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:55:50.403934 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:55:50.549088 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:55:50.549054 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:55:51.928174 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:55:51.928139 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 17:56:00.432442 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:00.432406 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:00.641172 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:00.641132 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:00.819109 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:00.819070 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:01.750033 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.750001 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8"] Oct 30 17:56:01.750404 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.750248 4603 cpu_manager.go:414] "containerMap: removing container" podUID="21a54420-0e75-4d4b-93cd-fe72c0cc2070" containerName="registry" Oct 30 17:56:01.750404 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.750259 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a54420-0e75-4d4b-93cd-fe72c0cc2070" containerName="registry" Oct 30 17:56:01.750404 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.750279 4603 cpu_manager.go:414] "containerMap: removing container" podUID="9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63" containerName="registry" Oct 30 17:56:01.750404 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.750284 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63" containerName="registry" Oct 30 17:56:01.750404 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.750346 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="21a54420-0e75-4d4b-93cd-fe72c0cc2070" containerName="registry" Oct 30 17:56:01.750404 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.750356 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b06cbb4-e7da-4fc8-b68f-a54ce08d8c63" containerName="registry" Oct 30 17:56:01.753121 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.753107 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8" Oct 30 17:56:01.789111 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.789087 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 17:56:01.789222 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.789118 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 17:56:01.789222 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.789172 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-2ncdz\"" Oct 30 17:56:01.811889 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.811870 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8"] Oct 30 17:56:01.893249 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.893227 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52"] Oct 30 17:56:01.896407 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.896392 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52" Oct 30 17:56:01.898111 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.898088 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acfe93e7-e9ce-4de6-a26a-e2b04f80bac9-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8\" (UID: \"acfe93e7-e9ce-4de6-a26a-e2b04f80bac9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8" Oct 30 17:56:01.898237 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.898159 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7cxn\" (UniqueName: \"kubernetes.io/projected/acfe93e7-e9ce-4de6-a26a-e2b04f80bac9-kube-api-access-p7cxn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8\" (UID: \"acfe93e7-e9ce-4de6-a26a-e2b04f80bac9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8" Oct 30 17:56:01.898289 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.898234 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acfe93e7-e9ce-4de6-a26a-e2b04f80bac9-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8\" (UID: \"acfe93e7-e9ce-4de6-a26a-e2b04f80bac9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8" Oct 30 17:56:01.910923 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.910906 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-7jmzb\"" Oct 30 17:56:01.922144 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.922124 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52"] Oct 30 17:56:01.999144 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.999125 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae1825ae-d8bb-482b-abe5-c119554865fb-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52\" (UID: \"ae1825ae-d8bb-482b-abe5-c119554865fb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52" Oct 30 17:56:01.999243 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.999152 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acfe93e7-e9ce-4de6-a26a-e2b04f80bac9-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8\" (UID: \"acfe93e7-e9ce-4de6-a26a-e2b04f80bac9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8" Oct 30 17:56:01.999243 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.999170 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae1825ae-d8bb-482b-abe5-c119554865fb-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52\" (UID: \"ae1825ae-d8bb-482b-abe5-c119554865fb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52" Oct 30 17:56:01.999313 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.999282 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tz8d\" (UniqueName: \"kubernetes.io/projected/ae1825ae-d8bb-482b-abe5-c119554865fb-kube-api-access-9tz8d\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52\" (UID: \"ae1825ae-d8bb-482b-abe5-c119554865fb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52" Oct 30 17:56:01.999382 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.999368 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7cxn\" (UniqueName: \"kubernetes.io/projected/acfe93e7-e9ce-4de6-a26a-e2b04f80bac9-kube-api-access-p7cxn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8\" (UID: \"acfe93e7-e9ce-4de6-a26a-e2b04f80bac9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8" Oct 30 17:56:01.999432 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.999394 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acfe93e7-e9ce-4de6-a26a-e2b04f80bac9-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8\" (UID: \"acfe93e7-e9ce-4de6-a26a-e2b04f80bac9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8" Oct 30 17:56:01.999494 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.999477 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acfe93e7-e9ce-4de6-a26a-e2b04f80bac9-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8\" (UID: \"acfe93e7-e9ce-4de6-a26a-e2b04f80bac9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8" Oct 30 17:56:01.999671 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:01.999655 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acfe93e7-e9ce-4de6-a26a-e2b04f80bac9-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8\" (UID: \"acfe93e7-e9ce-4de6-a26a-e2b04f80bac9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8" Oct 30 17:56:02.022871 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.022854 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7cxn\" (UniqueName: \"kubernetes.io/projected/acfe93e7-e9ce-4de6-a26a-e2b04f80bac9-kube-api-access-p7cxn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8\" (UID: \"acfe93e7-e9ce-4de6-a26a-e2b04f80bac9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8" Oct 30 17:56:02.067790 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.067771 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8" Oct 30 17:56:02.100778 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.100749 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae1825ae-d8bb-482b-abe5-c119554865fb-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52\" (UID: \"ae1825ae-d8bb-482b-abe5-c119554865fb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52" Oct 30 17:56:02.100918 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.100784 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae1825ae-d8bb-482b-abe5-c119554865fb-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52\" (UID: \"ae1825ae-d8bb-482b-abe5-c119554865fb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52" Oct 30 17:56:02.100918 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.100814 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tz8d\" (UniqueName: \"kubernetes.io/projected/ae1825ae-d8bb-482b-abe5-c119554865fb-kube-api-access-9tz8d\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52\" (UID: \"ae1825ae-d8bb-482b-abe5-c119554865fb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52" Oct 30 17:56:02.101080 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.101063 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae1825ae-d8bb-482b-abe5-c119554865fb-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52\" (UID: \"ae1825ae-d8bb-482b-abe5-c119554865fb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52" Oct 30 17:56:02.101150 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.101138 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae1825ae-d8bb-482b-abe5-c119554865fb-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52\" (UID: \"ae1825ae-d8bb-482b-abe5-c119554865fb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52" Oct 30 17:56:02.125264 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.125206 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tz8d\" (UniqueName: \"kubernetes.io/projected/ae1825ae-d8bb-482b-abe5-c119554865fb-kube-api-access-9tz8d\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52\" (UID: \"ae1825ae-d8bb-482b-abe5-c119554865fb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52" Oct 30 17:56:02.211856 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.211802 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52" Oct 30 17:56:02.291691 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.291618 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg"] Oct 30 17:56:02.295446 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.295425 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:02.310927 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.310904 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 17:56:02.311065 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.310941 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 17:56:02.311065 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.310972 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 17:56:02.311236 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.311155 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 17:56:02.311292 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.311285 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 17:56:02.311364 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.311352 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 17:56:02.311399 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.311353 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-44kjj\"" Oct 30 17:56:02.326568 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.326546 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg"] Oct 30 17:56:02.367401 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.367365 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:02.374784 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.374676 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8"] Oct 30 17:56:02.377346 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:56:02.377310 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacfe93e7_e9ce_4de6_a26a_e2b04f80bac9.slice/crio-94e3ae7cc950943723bb92a7600b3a91fff990aa15dc32cd6789f95a059d9355 WatchSource:0}: Error finding container 94e3ae7cc950943723bb92a7600b3a91fff990aa15dc32cd6789f95a059d9355: Status 404 returned error can't find the container with id 94e3ae7cc950943723bb92a7600b3a91fff990aa15dc32cd6789f95a059d9355 Oct 30 17:56:02.402573 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.402554 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/284f0ec5-9302-475d-95d4-e9a5304ed34c-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-zkxdg\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:02.402663 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.402584 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pdh2\" (UniqueName: \"kubernetes.io/projected/284f0ec5-9302-475d-95d4-e9a5304ed34c-kube-api-access-2pdh2\") pod \"istiod-openshift-gateway-7876bfcb74-zkxdg\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:02.402663 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.402604 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-zkxdg\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:02.402663 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.402622 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/284f0ec5-9302-475d-95d4-e9a5304ed34c-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-zkxdg\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:02.402782 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.402732 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-zkxdg\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:02.402782 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.402776 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-zkxdg\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:02.402859 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.402795 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-zkxdg\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:02.503301 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.503273 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-zkxdg\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:02.503396 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.503302 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-zkxdg\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:02.503396 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.503327 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/284f0ec5-9302-475d-95d4-e9a5304ed34c-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-zkxdg\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:02.503396 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.503359 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pdh2\" (UniqueName: \"kubernetes.io/projected/284f0ec5-9302-475d-95d4-e9a5304ed34c-kube-api-access-2pdh2\") pod \"istiod-openshift-gateway-7876bfcb74-zkxdg\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:02.503396 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.503386 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-zkxdg\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:02.503609 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.503412 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/284f0ec5-9302-475d-95d4-e9a5304ed34c-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-zkxdg\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:02.503609 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.503467 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-zkxdg\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:02.504018 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.503997 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-zkxdg\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:02.505897 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.505874 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/284f0ec5-9302-475d-95d4-e9a5304ed34c-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-zkxdg\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:02.506216 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.506194 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/284f0ec5-9302-475d-95d4-e9a5304ed34c-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-zkxdg\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:02.506302 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.506261 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-zkxdg\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:02.506361 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.506343 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-zkxdg\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:02.517035 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.517016 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52"] Oct 30 17:56:02.519900 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:56:02.519876 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1825ae_d8bb_482b_abe5_c119554865fb.slice/crio-610b9cd24f18fbc8b362ce6c13423f429582a3400948fa1aeac54e976ac1c940 WatchSource:0}: Error finding container 610b9cd24f18fbc8b362ce6c13423f429582a3400948fa1aeac54e976ac1c940: Status 404 returned error can't find the container with id 610b9cd24f18fbc8b362ce6c13423f429582a3400948fa1aeac54e976ac1c940 Oct 30 17:56:02.528489 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.528468 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-zkxdg\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:02.528789 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.528768 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pdh2\" (UniqueName: \"kubernetes.io/projected/284f0ec5-9302-475d-95d4-e9a5304ed34c-kube-api-access-2pdh2\") pod \"istiod-openshift-gateway-7876bfcb74-zkxdg\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:02.532190 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.532168 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:02.607126 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.607076 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:02.920574 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:02.920483 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg"] Oct 30 17:56:02.923927 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:56:02.923892 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod284f0ec5_9302_475d_95d4_e9a5304ed34c.slice/crio-01a1791979e5880a4aa404dced00cc0b61594a5b07c51a3b1aff7c9d8e0881c3 WatchSource:0}: Error finding container 01a1791979e5880a4aa404dced00cc0b61594a5b07c51a3b1aff7c9d8e0881c3: Status 404 returned error can't find the container with id 01a1791979e5880a4aa404dced00cc0b61594a5b07c51a3b1aff7c9d8e0881c3 Oct 30 17:56:03.227987 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:03.227905 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52" event={"ID":"ae1825ae-d8bb-482b-abe5-c119554865fb","Type":"ContainerStarted","Data":"610b9cd24f18fbc8b362ce6c13423f429582a3400948fa1aeac54e976ac1c940"} Oct 30 17:56:03.229934 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:03.229904 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8" event={"ID":"acfe93e7-e9ce-4de6-a26a-e2b04f80bac9","Type":"ContainerStarted","Data":"94e3ae7cc950943723bb92a7600b3a91fff990aa15dc32cd6789f95a059d9355"} Oct 30 17:56:03.231654 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:03.231622 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" event={"ID":"284f0ec5-9302-475d-95d4-e9a5304ed34c","Type":"ContainerStarted","Data":"01a1791979e5880a4aa404dced00cc0b61594a5b07c51a3b1aff7c9d8e0881c3"} Oct 30 17:56:03.731416 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:03.731353 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:05.845127 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:05.845093 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:56:05.845476 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:05.845177 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:56:06.246674 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:06.246592 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" event={"ID":"284f0ec5-9302-475d-95d4-e9a5304ed34c","Type":"ContainerStarted","Data":"35999d3bbaa647ec206b1486ffc7e7142ef39f1176e52e1d13c41239ce4df1cf"} Oct 30 17:56:06.248612 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:06.247637 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:06.248986 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:06.248957 4603 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-zkxdg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:56:06.249096 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:06.249009 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" podUID="284f0ec5-9302-475d-95d4-e9a5304ed34c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:56:06.316707 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:06.315677 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" podStartSLOduration=1.3976405650000001 podStartE2EDuration="4.315660263s" totalImagesPullingTime="2.918019698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:06.303832694 +0000 UTC m=+227.019402569" watchObservedRunningTime="2026-10-30 17:56:06.315660263 +0000 UTC m=+227.031230139" Oct 30 17:56:06.634081 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:06.634051 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:06.781751 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:06.781713 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:07.252850 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:07.252813 4603 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-zkxdg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:56:07.253320 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:07.252871 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" podUID="284f0ec5-9302-475d-95d4-e9a5304ed34c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:56:07.928134 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:07.928073 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:08.066036 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:08.066000 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:08.203916 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:08.203835 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:08.256808 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:08.256779 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:56:10.265204 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:10.265168 4603 generic.go:356] "Generic (PLEG): container finished" podID="ae1825ae-d8bb-482b-abe5-c119554865fb" containerID="d892338d917d151fa72cfc62b7ed9f491b08645c2ac56fb18aa51c1017408a74" exitCode=0 Oct 30 17:56:10.265606 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:10.265411 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52" event={"ID":"ae1825ae-d8bb-482b-abe5-c119554865fb","Type":"ContainerDied","Data":"d892338d917d151fa72cfc62b7ed9f491b08645c2ac56fb18aa51c1017408a74"} Oct 30 17:56:10.267132 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:10.267110 4603 generic.go:356] "Generic (PLEG): container finished" podID="acfe93e7-e9ce-4de6-a26a-e2b04f80bac9" containerID="761a739151f2cfd928af90d049e5fa2026282d83b84dc0cf378d54f42976a723" exitCode=0 Oct 30 17:56:10.267277 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:10.267180 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8" event={"ID":"acfe93e7-e9ce-4de6-a26a-e2b04f80bac9","Type":"ContainerDied","Data":"761a739151f2cfd928af90d049e5fa2026282d83b84dc0cf378d54f42976a723"} Oct 30 17:56:11.273108 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:11.273079 4603 generic.go:356] "Generic (PLEG): container finished" podID="ae1825ae-d8bb-482b-abe5-c119554865fb" containerID="c7cc2bbde9f46e4f6664d534382ff21b08a21f6c5a6cdb69db709f96ef2b2a5e" exitCode=0 Oct 30 17:56:11.273486 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:11.273144 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52" event={"ID":"ae1825ae-d8bb-482b-abe5-c119554865fb","Type":"ContainerDied","Data":"c7cc2bbde9f46e4f6664d534382ff21b08a21f6c5a6cdb69db709f96ef2b2a5e"} Oct 30 17:56:11.351593 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:11.351556 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:11.686071 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:11.686003 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:13.283443 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:13.283410 4603 generic.go:356] "Generic (PLEG): container finished" podID="acfe93e7-e9ce-4de6-a26a-e2b04f80bac9" containerID="2f92b3661ed77acc5785cfb3f9530b324444ae21a17ae79d15bd2a841e18e8e0" exitCode=0 Oct 30 17:56:13.283863 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:13.283447 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8" event={"ID":"acfe93e7-e9ce-4de6-a26a-e2b04f80bac9","Type":"ContainerDied","Data":"2f92b3661ed77acc5785cfb3f9530b324444ae21a17ae79d15bd2a841e18e8e0"} Oct 30 17:56:18.307403 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:18.307366 4603 generic.go:356] "Generic (PLEG): container finished" podID="ae1825ae-d8bb-482b-abe5-c119554865fb" containerID="11cef164a5d32c1bcb29df506705a6d2b4f3e52d3eb45ae96cf87a4e42b0b919" exitCode=0 Oct 30 17:56:18.307833 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:18.307445 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52" event={"ID":"ae1825ae-d8bb-482b-abe5-c119554865fb","Type":"ContainerDied","Data":"11cef164a5d32c1bcb29df506705a6d2b4f3e52d3eb45ae96cf87a4e42b0b919"} Oct 30 17:56:19.454225 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:19.454201 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52" Oct 30 17:56:19.542227 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:19.542201 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1825ae-d8bb-482b-abe5-c119554865fb-util\" (UniqueName: \"kubernetes.io/empty-dir/ae1825ae-d8bb-482b-abe5-c119554865fb-util\") pod \"ae1825ae-d8bb-482b-abe5-c119554865fb\" (UID: \"ae1825ae-d8bb-482b-abe5-c119554865fb\") " Oct 30 17:56:19.542403 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:19.542295 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1825ae-d8bb-482b-abe5-c119554865fb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae1825ae-d8bb-482b-abe5-c119554865fb-bundle\") pod \"ae1825ae-d8bb-482b-abe5-c119554865fb\" (UID: \"ae1825ae-d8bb-482b-abe5-c119554865fb\") " Oct 30 17:56:19.542403 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:19.542368 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1825ae-d8bb-482b-abe5-c119554865fb-kube-api-access-9tz8d\" (UniqueName: \"kubernetes.io/projected/ae1825ae-d8bb-482b-abe5-c119554865fb-kube-api-access-9tz8d\") pod \"ae1825ae-d8bb-482b-abe5-c119554865fb\" (UID: \"ae1825ae-d8bb-482b-abe5-c119554865fb\") " Oct 30 17:56:19.542948 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:19.542915 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1825ae-d8bb-482b-abe5-c119554865fb-bundle" pod "ae1825ae-d8bb-482b-abe5-c119554865fb" (UID: "ae1825ae-d8bb-482b-abe5-c119554865fb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:19.545141 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:19.545115 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1825ae-d8bb-482b-abe5-c119554865fb-kube-api-access-9tz8d" pod "ae1825ae-d8bb-482b-abe5-c119554865fb" (UID: "ae1825ae-d8bb-482b-abe5-c119554865fb"). InnerVolumeSpecName "kube-api-access-9tz8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:56:19.547437 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:19.547411 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1825ae-d8bb-482b-abe5-c119554865fb-util" pod "ae1825ae-d8bb-482b-abe5-c119554865fb" (UID: "ae1825ae-d8bb-482b-abe5-c119554865fb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:19.643056 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:19.643002 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tz8d\" (UniqueName: \"kubernetes.io/projected/ae1825ae-d8bb-482b-abe5-c119554865fb-kube-api-access-9tz8d\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:56:19.643056 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:19.643025 4603 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae1825ae-d8bb-482b-abe5-c119554865fb-util\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:56:19.643056 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:19.643035 4603 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae1825ae-d8bb-482b-abe5-c119554865fb-bundle\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:56:20.316623 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:20.316588 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52" event={"ID":"ae1825ae-d8bb-482b-abe5-c119554865fb","Type":"ContainerDied","Data":"610b9cd24f18fbc8b362ce6c13423f429582a3400948fa1aeac54e976ac1c940"} Oct 30 17:56:20.316623 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:20.316611 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j6g52" Oct 30 17:56:20.316844 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:20.316639 4603 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="610b9cd24f18fbc8b362ce6c13423f429582a3400948fa1aeac54e976ac1c940" Oct 30 17:56:20.318871 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:20.318841 4603 generic.go:356] "Generic (PLEG): container finished" podID="acfe93e7-e9ce-4de6-a26a-e2b04f80bac9" containerID="6264228ec90b01d36b7395b9253366cda766175f4b5ceca2e82d466e30d53f8f" exitCode=0 Oct 30 17:56:20.318988 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:20.318881 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8" event={"ID":"acfe93e7-e9ce-4de6-a26a-e2b04f80bac9","Type":"ContainerDied","Data":"6264228ec90b01d36b7395b9253366cda766175f4b5ceca2e82d466e30d53f8f"} Oct 30 17:56:20.962398 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:20.962365 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:21.101171 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:21.101139 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:21.441330 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:21.441308 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8" Oct 30 17:56:21.558238 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:21.558209 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acfe93e7-e9ce-4de6-a26a-e2b04f80bac9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/acfe93e7-e9ce-4de6-a26a-e2b04f80bac9-bundle\") pod \"acfe93e7-e9ce-4de6-a26a-e2b04f80bac9\" (UID: \"acfe93e7-e9ce-4de6-a26a-e2b04f80bac9\") " Oct 30 17:56:21.558406 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:21.558251 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acfe93e7-e9ce-4de6-a26a-e2b04f80bac9-kube-api-access-p7cxn\" (UniqueName: \"kubernetes.io/projected/acfe93e7-e9ce-4de6-a26a-e2b04f80bac9-kube-api-access-p7cxn\") pod \"acfe93e7-e9ce-4de6-a26a-e2b04f80bac9\" (UID: \"acfe93e7-e9ce-4de6-a26a-e2b04f80bac9\") " Oct 30 17:56:21.558406 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:21.558294 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acfe93e7-e9ce-4de6-a26a-e2b04f80bac9-util\" (UniqueName: \"kubernetes.io/empty-dir/acfe93e7-e9ce-4de6-a26a-e2b04f80bac9-util\") pod \"acfe93e7-e9ce-4de6-a26a-e2b04f80bac9\" (UID: \"acfe93e7-e9ce-4de6-a26a-e2b04f80bac9\") " Oct 30 17:56:21.558945 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:21.558910 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acfe93e7-e9ce-4de6-a26a-e2b04f80bac9-bundle" pod "acfe93e7-e9ce-4de6-a26a-e2b04f80bac9" (UID: "acfe93e7-e9ce-4de6-a26a-e2b04f80bac9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:21.560525 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:21.560472 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acfe93e7-e9ce-4de6-a26a-e2b04f80bac9-kube-api-access-p7cxn" pod "acfe93e7-e9ce-4de6-a26a-e2b04f80bac9" (UID: "acfe93e7-e9ce-4de6-a26a-e2b04f80bac9"). InnerVolumeSpecName "kube-api-access-p7cxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:56:21.562585 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:21.562565 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acfe93e7-e9ce-4de6-a26a-e2b04f80bac9-util" pod "acfe93e7-e9ce-4de6-a26a-e2b04f80bac9" (UID: "acfe93e7-e9ce-4de6-a26a-e2b04f80bac9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:21.659245 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:21.659180 4603 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acfe93e7-e9ce-4de6-a26a-e2b04f80bac9-bundle\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:56:21.659245 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:21.659206 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7cxn\" (UniqueName: \"kubernetes.io/projected/acfe93e7-e9ce-4de6-a26a-e2b04f80bac9-kube-api-access-p7cxn\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:56:21.659245 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:21.659219 4603 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acfe93e7-e9ce-4de6-a26a-e2b04f80bac9-util\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:56:21.927963 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:56:21.927890 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 17:56:22.327615 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:22.327582 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8" event={"ID":"acfe93e7-e9ce-4de6-a26a-e2b04f80bac9","Type":"ContainerDied","Data":"94e3ae7cc950943723bb92a7600b3a91fff990aa15dc32cd6789f95a059d9355"} Oct 30 17:56:22.327985 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:22.327632 4603 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94e3ae7cc950943723bb92a7600b3a91fff990aa15dc32cd6789f95a059d9355" Oct 30 17:56:22.327985 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:22.327594 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j8v8" Oct 30 17:56:31.349701 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:31.349660 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:31.491882 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:31.491852 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:32.091964 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.091935 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:32.231851 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.231823 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x"] Oct 30 17:56:32.232081 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.232069 4603 cpu_manager.go:414] "containerMap: removing container" podUID="acfe93e7-e9ce-4de6-a26a-e2b04f80bac9" containerName="util" Oct 30 17:56:32.232127 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.232082 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acfe93e7-e9ce-4de6-a26a-e2b04f80bac9" containerName="util" Oct 30 17:56:32.232127 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.232095 4603 cpu_manager.go:414] "containerMap: removing container" podUID="acfe93e7-e9ce-4de6-a26a-e2b04f80bac9" containerName="pull" Oct 30 17:56:32.232127 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.232100 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acfe93e7-e9ce-4de6-a26a-e2b04f80bac9" containerName="pull" Oct 30 17:56:32.232127 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.232109 4603 cpu_manager.go:414] "containerMap: removing container" podUID="ae1825ae-d8bb-482b-abe5-c119554865fb" containerName="extract" Oct 30 17:56:32.232127 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.232115 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae1825ae-d8bb-482b-abe5-c119554865fb" containerName="extract" Oct 30 17:56:32.232289 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.232131 4603 cpu_manager.go:414] "containerMap: removing container" podUID="ae1825ae-d8bb-482b-abe5-c119554865fb" containerName="pull" Oct 30 17:56:32.232289 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.232136 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae1825ae-d8bb-482b-abe5-c119554865fb" containerName="pull" Oct 30 17:56:32.232289 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.232143 4603 cpu_manager.go:414] "containerMap: removing container" podUID="acfe93e7-e9ce-4de6-a26a-e2b04f80bac9" containerName="extract" Oct 30 17:56:32.232289 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.232147 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acfe93e7-e9ce-4de6-a26a-e2b04f80bac9" containerName="extract" Oct 30 17:56:32.232289 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.232153 4603 cpu_manager.go:414] "containerMap: removing container" podUID="ae1825ae-d8bb-482b-abe5-c119554865fb" containerName="util" Oct 30 17:56:32.232289 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.232157 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae1825ae-d8bb-482b-abe5-c119554865fb" containerName="util" Oct 30 17:56:32.232289 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.232202 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="acfe93e7-e9ce-4de6-a26a-e2b04f80bac9" containerName="extract" Oct 30 17:56:32.232289 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.232209 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae1825ae-d8bb-482b-abe5-c119554865fb" containerName="extract" Oct 30 17:56:32.234015 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.234001 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" Oct 30 17:56:32.270033 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.270003 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x"] Oct 30 17:56:32.322023 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.322001 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 17:56:32.322144 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.322020 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 17:56:32.322672 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.322653 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 17:56:32.322718 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.322669 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 17:56:32.322820 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.322806 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 17:56:32.322856 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.322826 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-v2gs7\"" Oct 30 17:56:32.329633 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.329611 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a-webhook-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-rpf9x\" (UID: \"6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" Oct 30 17:56:32.329745 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.329655 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a-metrics-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-rpf9x\" (UID: \"6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" Oct 30 17:56:32.329745 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.329683 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-rpf9x\" (UID: \"6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" Oct 30 17:56:32.329745 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.329733 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98bvk\" (UniqueName: \"kubernetes.io/projected/6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a-kube-api-access-98bvk\") pod \"opendatahub-operator-controller-manager-64bcb8d955-rpf9x\" (UID: \"6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" Oct 30 17:56:32.329843 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.329791 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a-manifests\") pod \"opendatahub-operator-controller-manager-64bcb8d955-rpf9x\" (UID: \"6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" Oct 30 17:56:32.329843 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.329811 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a-tmp\") pod \"opendatahub-operator-controller-manager-64bcb8d955-rpf9x\" (UID: \"6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" Oct 30 17:56:32.372490 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.372427 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:32.430089 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.430066 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a-webhook-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-rpf9x\" (UID: \"6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" Oct 30 17:56:32.430193 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.430099 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a-metrics-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-rpf9x\" (UID: \"6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" Oct 30 17:56:32.430193 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.430136 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-rpf9x\" (UID: \"6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" Oct 30 17:56:32.430193 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.430172 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98bvk\" (UniqueName: \"kubernetes.io/projected/6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a-kube-api-access-98bvk\") pod \"opendatahub-operator-controller-manager-64bcb8d955-rpf9x\" (UID: \"6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" Oct 30 17:56:32.430352 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.430221 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a-manifests\") pod \"opendatahub-operator-controller-manager-64bcb8d955-rpf9x\" (UID: \"6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" Oct 30 17:56:32.430352 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.430251 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a-tmp\") pod \"opendatahub-operator-controller-manager-64bcb8d955-rpf9x\" (UID: \"6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" Oct 30 17:56:32.430739 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.430715 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a-manifests\") pod \"opendatahub-operator-controller-manager-64bcb8d955-rpf9x\" (UID: \"6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" Oct 30 17:56:32.432647 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.432628 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a-tmp\") pod \"opendatahub-operator-controller-manager-64bcb8d955-rpf9x\" (UID: \"6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" Oct 30 17:56:32.432746 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.432692 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-rpf9x\" (UID: \"6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" Oct 30 17:56:32.432897 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.432880 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a-webhook-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-rpf9x\" (UID: \"6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" Oct 30 17:56:32.432960 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.432942 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a-metrics-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-rpf9x\" (UID: \"6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" Oct 30 17:56:32.453387 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.453361 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98bvk\" (UniqueName: \"kubernetes.io/projected/6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a-kube-api-access-98bvk\") pod \"opendatahub-operator-controller-manager-64bcb8d955-rpf9x\" (UID: \"6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" Oct 30 17:56:32.517857 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.517832 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:32.545982 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.545959 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" Oct 30 17:56:32.676024 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.675957 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:32.903824 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:32.903787 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x"] Oct 30 17:56:32.906733 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:56:32.906697 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6c8d89_f2c3_40b3_bd3e_58f0def5db1a.slice/crio-e9ab560622ce2a14bb4ea2642aa00d6256faf5b5d42eb0a80e05a1c6897a7ee5 WatchSource:0}: Error finding container e9ab560622ce2a14bb4ea2642aa00d6256faf5b5d42eb0a80e05a1c6897a7ee5: Status 404 returned error can't find the container with id e9ab560622ce2a14bb4ea2642aa00d6256faf5b5d42eb0a80e05a1c6897a7ee5 Oct 30 17:56:33.367791 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:33.367746 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" event={"ID":"6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a","Type":"ContainerStarted","Data":"e9ab560622ce2a14bb4ea2642aa00d6256faf5b5d42eb0a80e05a1c6897a7ee5"} Oct 30 17:56:33.381627 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:33.381593 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:35.787093 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:35.787063 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:35.955920 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:35.955854 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:36.350514 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:36.350467 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:36.381550 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:36.381522 4603 generic.go:356] "Generic (PLEG): container finished" podID="6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a" containerID="3bd062ac85411a1d238d0fce36fef6099c6feff7b940232593ff6102f842ec8d" exitCode=0 Oct 30 17:56:36.381674 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:36.381557 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" event={"ID":"6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a","Type":"ContainerDied","Data":"3bd062ac85411a1d238d0fce36fef6099c6feff7b940232593ff6102f842ec8d"} Oct 30 17:56:37.146747 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:37.146715 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:37.388077 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:37.388040 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" event={"ID":"6b6c8d89-f2c3-40b3-bd3e-58f0def5db1a","Type":"ContainerStarted","Data":"74b409e36b727806dc4eecc577759ee707cc1bf04c8d64c2bd38de3989c0deac"} Oct 30 17:56:37.388237 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:37.388192 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" Oct 30 17:56:37.475463 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:37.475382 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" podStartSLOduration=2.006124969 podStartE2EDuration="5.475370263s" totalImagesPullingTime="3.048489534s" totalInitContainerRuntime="420.75576ms" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:37.463487256 +0000 UTC m=+258.179057129" watchObservedRunningTime="2026-10-30 17:56:37.475370263 +0000 UTC m=+258.190940136" Oct 30 17:56:37.913567 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:37.913539 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:38.567299 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:38.567256 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:38.716792 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:38.716763 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:38.943892 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:38.943802 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:43.843020 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:43.842986 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:44.884032 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:44.883998 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:45.545548 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:45.545495 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:45.724637 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:45.724600 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:46.535803 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:46.535772 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:46.692188 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:46.692159 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:47.575619 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:47.575584 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:47.728168 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:47.728131 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:48.200788 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:48.200755 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:48.393998 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:48.393972 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-rpf9x" Oct 30 17:56:50.829449 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:50.829412 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:51.927669 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:56:51.927638 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 17:56:56.637062 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:56.637033 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:56.810088 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:56.810056 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:57.834031 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:57.833988 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:58.492090 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:58.492057 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:58.844235 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:58.844201 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:59.113089 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:59.112997 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:59.431521 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:56:59.431406 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:00.921789 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:00.921756 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-qfgcb"] Oct 30 17:57:00.923958 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:00.923941 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-qfgcb" Oct 30 17:57:00.939737 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:00.939719 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 17:57:00.951366 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:00.951337 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 17:57:00.951491 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:00.951448 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 17:57:00.951581 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:00.951490 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xzxsc\"" Oct 30 17:57:00.952482 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:00.952456 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-qfgcb"] Oct 30 17:57:00.956006 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:00.955981 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:01.024755 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:01.024728 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b9d8afd-1ece-4b82-a62d-b47d6cf5e10b-plugin-serving-cert\") pod \"kuadrant-console-plugin-7bbc6bb9d7-qfgcb\" (UID: \"2b9d8afd-1ece-4b82-a62d-b47d6cf5e10b\") " pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-qfgcb" Oct 30 17:57:01.024863 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:01.024789 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5xbv\" (UniqueName: \"kubernetes.io/projected/2b9d8afd-1ece-4b82-a62d-b47d6cf5e10b-kube-api-access-s5xbv\") pod \"kuadrant-console-plugin-7bbc6bb9d7-qfgcb\" (UID: \"2b9d8afd-1ece-4b82-a62d-b47d6cf5e10b\") " pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-qfgcb" Oct 30 17:57:01.101692 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:01.101665 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:01.125799 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:01.125775 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xbv\" (UniqueName: \"kubernetes.io/projected/2b9d8afd-1ece-4b82-a62d-b47d6cf5e10b-kube-api-access-s5xbv\") pod \"kuadrant-console-plugin-7bbc6bb9d7-qfgcb\" (UID: \"2b9d8afd-1ece-4b82-a62d-b47d6cf5e10b\") " pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-qfgcb" Oct 30 17:57:01.125893 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:01.125814 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b9d8afd-1ece-4b82-a62d-b47d6cf5e10b-plugin-serving-cert\") pod \"kuadrant-console-plugin-7bbc6bb9d7-qfgcb\" (UID: \"2b9d8afd-1ece-4b82-a62d-b47d6cf5e10b\") " pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-qfgcb" Oct 30 17:57:01.128372 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:01.128345 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b9d8afd-1ece-4b82-a62d-b47d6cf5e10b-plugin-serving-cert\") pod \"kuadrant-console-plugin-7bbc6bb9d7-qfgcb\" (UID: \"2b9d8afd-1ece-4b82-a62d-b47d6cf5e10b\") " pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-qfgcb" Oct 30 17:57:01.152895 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:01.152869 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5xbv\" (UniqueName: \"kubernetes.io/projected/2b9d8afd-1ece-4b82-a62d-b47d6cf5e10b-kube-api-access-s5xbv\") pod \"kuadrant-console-plugin-7bbc6bb9d7-qfgcb\" (UID: \"2b9d8afd-1ece-4b82-a62d-b47d6cf5e10b\") " pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-qfgcb" Oct 30 17:57:01.251953 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:01.251903 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-qfgcb" Oct 30 17:57:01.547164 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:01.547131 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-qfgcb"] Oct 30 17:57:01.550624 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:57:01.550597 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b9d8afd_1ece_4b82_a62d_b47d6cf5e10b.slice/crio-56ddf49407a9fb5dcdef5a0d814facd94206a478a63dcd79392957b8a12f822a WatchSource:0}: Error finding container 56ddf49407a9fb5dcdef5a0d814facd94206a478a63dcd79392957b8a12f822a: Status 404 returned error can't find the container with id 56ddf49407a9fb5dcdef5a0d814facd94206a478a63dcd79392957b8a12f822a Oct 30 17:57:02.483827 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:02.483782 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-qfgcb" event={"ID":"2b9d8afd-1ece-4b82-a62d-b47d6cf5e10b","Type":"ContainerStarted","Data":"56ddf49407a9fb5dcdef5a0d814facd94206a478a63dcd79392957b8a12f822a"} Oct 30 17:57:02.993944 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:02.993900 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:03.178577 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:03.178529 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:03.383916 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:03.383884 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:03.737176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:03.737108 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-9c55b8cc4-7pfgm"] Oct 30 17:57:03.745747 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:03.745622 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-7pfgm" Oct 30 17:57:03.768993 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:03.768956 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:03.773096 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:03.773064 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9c55b8cc4-7pfgm"] Oct 30 17:57:03.849817 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:03.849785 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p48tt\" (UniqueName: \"kubernetes.io/projected/e8ba7b34-3707-4350-b3c0-983d6afc6fd3-kube-api-access-p48tt\") pod \"kube-auth-proxy-9c55b8cc4-7pfgm\" (UID: \"e8ba7b34-3707-4350-b3c0-983d6afc6fd3\") " pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-7pfgm" Oct 30 17:57:03.849977 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:03.849874 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8ba7b34-3707-4350-b3c0-983d6afc6fd3-tls-certs\") pod \"kube-auth-proxy-9c55b8cc4-7pfgm\" (UID: \"e8ba7b34-3707-4350-b3c0-983d6afc6fd3\") " pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-7pfgm" Oct 30 17:57:03.849977 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:03.849941 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8ba7b34-3707-4350-b3c0-983d6afc6fd3-tmp\") pod \"kube-auth-proxy-9c55b8cc4-7pfgm\" (UID: \"e8ba7b34-3707-4350-b3c0-983d6afc6fd3\") " pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-7pfgm" Oct 30 17:57:03.856254 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:03.856226 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 17:57:03.856391 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:03.856273 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-bn27s\"" Oct 30 17:57:03.856391 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:03.856303 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 17:57:03.950360 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:03.950333 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8ba7b34-3707-4350-b3c0-983d6afc6fd3-tls-certs\") pod \"kube-auth-proxy-9c55b8cc4-7pfgm\" (UID: \"e8ba7b34-3707-4350-b3c0-983d6afc6fd3\") " pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-7pfgm" Oct 30 17:57:03.950470 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:03.950383 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8ba7b34-3707-4350-b3c0-983d6afc6fd3-tmp\") pod \"kube-auth-proxy-9c55b8cc4-7pfgm\" (UID: \"e8ba7b34-3707-4350-b3c0-983d6afc6fd3\") " pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-7pfgm" Oct 30 17:57:03.950470 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:03.950463 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48tt\" (UniqueName: \"kubernetes.io/projected/e8ba7b34-3707-4350-b3c0-983d6afc6fd3-kube-api-access-p48tt\") pod \"kube-auth-proxy-9c55b8cc4-7pfgm\" (UID: \"e8ba7b34-3707-4350-b3c0-983d6afc6fd3\") " pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-7pfgm" Oct 30 17:57:03.953013 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:03.952989 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8ba7b34-3707-4350-b3c0-983d6afc6fd3-tmp\") pod \"kube-auth-proxy-9c55b8cc4-7pfgm\" (UID: \"e8ba7b34-3707-4350-b3c0-983d6afc6fd3\") " pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-7pfgm" Oct 30 17:57:03.953448 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:03.953428 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8ba7b34-3707-4350-b3c0-983d6afc6fd3-tls-certs\") pod \"kube-auth-proxy-9c55b8cc4-7pfgm\" (UID: \"e8ba7b34-3707-4350-b3c0-983d6afc6fd3\") " pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-7pfgm" Oct 30 17:57:03.974415 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:03.974393 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p48tt\" (UniqueName: \"kubernetes.io/projected/e8ba7b34-3707-4350-b3c0-983d6afc6fd3-kube-api-access-p48tt\") pod \"kube-auth-proxy-9c55b8cc4-7pfgm\" (UID: \"e8ba7b34-3707-4350-b3c0-983d6afc6fd3\") " pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-7pfgm" Oct 30 17:57:04.038035 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:04.038001 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:04.058963 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:04.058939 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-7pfgm" Oct 30 17:57:04.195051 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:04.195017 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:04.354688 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:04.354662 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9c55b8cc4-7pfgm"] Oct 30 17:57:04.357703 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:57:04.357674 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ba7b34_3707_4350_b3c0_983d6afc6fd3.slice/crio-eda42a3e55ef780f56b84c75436bbbb0f6353eb7e46e69cc2464dce03c6e5f96 WatchSource:0}: Error finding container eda42a3e55ef780f56b84c75436bbbb0f6353eb7e46e69cc2464dce03c6e5f96: Status 404 returned error can't find the container with id eda42a3e55ef780f56b84c75436bbbb0f6353eb7e46e69cc2464dce03c6e5f96 Oct 30 17:57:04.379198 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:04.379166 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:04.492670 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:04.492645 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-qfgcb" event={"ID":"2b9d8afd-1ece-4b82-a62d-b47d6cf5e10b","Type":"ContainerStarted","Data":"be17b99ea041193f75c25d117b64140834f360b640627483ba4fe972c5fc01db"} Oct 30 17:57:04.493856 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:04.493830 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-7pfgm" event={"ID":"e8ba7b34-3707-4350-b3c0-983d6afc6fd3","Type":"ContainerStarted","Data":"eda42a3e55ef780f56b84c75436bbbb0f6353eb7e46e69cc2464dce03c6e5f96"} Oct 30 17:57:04.531189 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:04.531152 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-qfgcb" podStartSLOduration=2.136479757 podStartE2EDuration="4.531140749s" totalImagesPullingTime="2.394660992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:04.530651957 +0000 UTC m=+285.246221830" watchObservedRunningTime="2026-10-30 17:57:04.531140749 +0000 UTC m=+285.246710601" Oct 30 17:57:05.373265 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:05.373217 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:05.518381 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:05.518319 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:05.661098 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:05.661015 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:06.437493 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:06.437456 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:06.586964 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:06.586931 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:07.509018 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.508986 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-7pfgm" event={"ID":"e8ba7b34-3707-4350-b3c0-983d6afc6fd3","Type":"ContainerStarted","Data":"6121eeb52ec818bd3329b704383708dbd5b9a6fddad5f089ed51fc4d6a13c430"} Oct 30 17:57:07.549664 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.549614 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-7pfgm" podStartSLOduration=2.17076021 podStartE2EDuration="4.549599288s" totalImagesPullingTime="2.378839078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:07.548733397 +0000 UTC m=+288.264303271" watchObservedRunningTime="2026-10-30 17:57:07.549599288 +0000 UTC m=+288.265169162" Oct 30 17:57:07.736013 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.735979 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-tf6tc"] Oct 30 17:57:07.739386 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.739366 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tf6tc" Oct 30 17:57:07.757658 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.757638 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 17:57:07.758263 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.758101 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-q8qh5\"" Oct 30 17:57:07.758263 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.758102 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 17:57:07.759847 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.759808 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 17:57:07.762866 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.762851 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-tf6tc"] Oct 30 17:57:07.762947 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.762922 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t" Oct 30 17:57:07.779782 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.779763 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/257e1e70-9c25-4ac1-b2e6-c2b2e2a8154d-config-file\") pod \"limitador-limitador-f7d6b5c7c-n4z9t\" (UID: \"257e1e70-9c25-4ac1-b2e6-c2b2e2a8154d\") " pod="kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t" Oct 30 17:57:07.779862 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.779794 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wll6w\" (UniqueName: \"kubernetes.io/projected/ef26451a-379c-4cd3-957e-1a61d3a1e55a-kube-api-access-wll6w\") pod \"cert-manager-cainjector-5c57bbd4fc-tf6tc\" (UID: \"ef26451a-379c-4cd3-957e-1a61d3a1e55a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tf6tc" Oct 30 17:57:07.779897 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.779884 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvw2p\" (UniqueName: \"kubernetes.io/projected/257e1e70-9c25-4ac1-b2e6-c2b2e2a8154d-kube-api-access-mvw2p\") pod \"limitador-limitador-f7d6b5c7c-n4z9t\" (UID: \"257e1e70-9c25-4ac1-b2e6-c2b2e2a8154d\") " pod="kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t" Oct 30 17:57:07.779964 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.779947 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef26451a-379c-4cd3-957e-1a61d3a1e55a-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-tf6tc\" (UID: \"ef26451a-379c-4cd3-957e-1a61d3a1e55a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tf6tc" Oct 30 17:57:07.785732 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.785708 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 17:57:07.835977 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.835952 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 17:57:07.848981 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.848961 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 17:57:07.881171 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.881148 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wll6w\" (UniqueName: \"kubernetes.io/projected/ef26451a-379c-4cd3-957e-1a61d3a1e55a-kube-api-access-wll6w\") pod \"cert-manager-cainjector-5c57bbd4fc-tf6tc\" (UID: \"ef26451a-379c-4cd3-957e-1a61d3a1e55a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tf6tc" Oct 30 17:57:07.881299 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.881202 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvw2p\" (UniqueName: \"kubernetes.io/projected/257e1e70-9c25-4ac1-b2e6-c2b2e2a8154d-kube-api-access-mvw2p\") pod \"limitador-limitador-f7d6b5c7c-n4z9t\" (UID: \"257e1e70-9c25-4ac1-b2e6-c2b2e2a8154d\") " pod="kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t" Oct 30 17:57:07.881299 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.881231 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef26451a-379c-4cd3-957e-1a61d3a1e55a-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-tf6tc\" (UID: \"ef26451a-379c-4cd3-957e-1a61d3a1e55a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tf6tc" Oct 30 17:57:07.881299 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.881256 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/257e1e70-9c25-4ac1-b2e6-c2b2e2a8154d-config-file\") pod \"limitador-limitador-f7d6b5c7c-n4z9t\" (UID: \"257e1e70-9c25-4ac1-b2e6-c2b2e2a8154d\") " pod="kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t" Oct 30 17:57:07.882225 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.881821 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/257e1e70-9c25-4ac1-b2e6-c2b2e2a8154d-config-file\") pod \"limitador-limitador-f7d6b5c7c-n4z9t\" (UID: \"257e1e70-9c25-4ac1-b2e6-c2b2e2a8154d\") " pod="kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t" Oct 30 17:57:07.903829 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.903809 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef26451a-379c-4cd3-957e-1a61d3a1e55a-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-tf6tc\" (UID: \"ef26451a-379c-4cd3-957e-1a61d3a1e55a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tf6tc" Oct 30 17:57:07.904026 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.904009 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvw2p\" (UniqueName: \"kubernetes.io/projected/257e1e70-9c25-4ac1-b2e6-c2b2e2a8154d-kube-api-access-mvw2p\") pod \"limitador-limitador-f7d6b5c7c-n4z9t\" (UID: \"257e1e70-9c25-4ac1-b2e6-c2b2e2a8154d\") " pod="kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t" Oct 30 17:57:07.904071 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.904016 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wll6w\" (UniqueName: \"kubernetes.io/projected/ef26451a-379c-4cd3-957e-1a61d3a1e55a-kube-api-access-wll6w\") pod \"cert-manager-cainjector-5c57bbd4fc-tf6tc\" (UID: \"ef26451a-379c-4cd3-957e-1a61d3a1e55a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tf6tc" Oct 30 17:57:07.949005 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:07.948981 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:08.050804 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:08.050777 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tf6tc" Oct 30 17:57:08.075200 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:08.075178 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t" Oct 30 17:57:08.361325 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:08.361243 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-tf6tc"] Oct 30 17:57:08.364149 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:57:08.364122 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef26451a_379c_4cd3_957e_1a61d3a1e55a.slice/crio-e3ae585fc0f2f053f2b035658f7492a511e0aafcb64adc9d51da967ab73e300e WatchSource:0}: Error finding container e3ae585fc0f2f053f2b035658f7492a511e0aafcb64adc9d51da967ab73e300e: Status 404 returned error can't find the container with id e3ae585fc0f2f053f2b035658f7492a511e0aafcb64adc9d51da967ab73e300e Oct 30 17:57:08.403910 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:08.403887 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 17:57:08.406372 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:57:08.406347 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257e1e70_9c25_4ac1_b2e6_c2b2e2a8154d.slice/crio-632afa85b9ea181b45f409ceb4cf8652a007678858440d908499fe429dd6144c WatchSource:0}: Error finding container 632afa85b9ea181b45f409ceb4cf8652a007678858440d908499fe429dd6144c: Status 404 returned error can't find the container with id 632afa85b9ea181b45f409ceb4cf8652a007678858440d908499fe429dd6144c Oct 30 17:57:08.514860 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:08.514821 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tf6tc" event={"ID":"ef26451a-379c-4cd3-957e-1a61d3a1e55a","Type":"ContainerStarted","Data":"e3ae585fc0f2f053f2b035658f7492a511e0aafcb64adc9d51da967ab73e300e"} Oct 30 17:57:08.516053 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:08.516018 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t" event={"ID":"257e1e70-9c25-4ac1-b2e6-c2b2e2a8154d","Type":"ContainerStarted","Data":"632afa85b9ea181b45f409ceb4cf8652a007678858440d908499fe429dd6144c"} Oct 30 17:57:11.531292 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:11.531250 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t" event={"ID":"257e1e70-9c25-4ac1-b2e6-c2b2e2a8154d","Type":"ContainerStarted","Data":"e9375381ac1838be0a9b3e7dc47e0328cd34514e8d253af00c6b7fcd61aa3452"} Oct 30 17:57:11.531811 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:11.531319 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t" Oct 30 17:57:11.532807 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:11.532775 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tf6tc" event={"ID":"ef26451a-379c-4cd3-957e-1a61d3a1e55a","Type":"ContainerStarted","Data":"b31e20b4f9f6d5228b42a235516f61b042cbb44a62d650c20cdd8956b2dcd64a"} Oct 30 17:57:11.571711 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:11.571656 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t" podStartSLOduration=1.9282765259999999 podStartE2EDuration="4.571643622s" totalImagesPullingTime="2.643367096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:11.571072988 +0000 UTC m=+292.286642862" watchObservedRunningTime="2026-10-30 17:57:11.571643622 +0000 UTC m=+292.287213490" Oct 30 17:57:11.609416 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:11.609362 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tf6tc" podStartSLOduration=2.018576197 podStartE2EDuration="4.60934966s" totalImagesPullingTime="2.590773463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:11.608915829 +0000 UTC m=+292.324485702" watchObservedRunningTime="2026-10-30 17:57:11.60934966 +0000 UTC m=+292.324919535" Oct 30 17:57:11.812259 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:11.812178 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:12.032726 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:12.032686 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:19.435222 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:19.435195 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 17:57:19.435860 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:19.435840 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 17:57:21.928471 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:57:21.928430 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 17:57:22.537923 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:22.537890 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t" Oct 30 17:57:33.915799 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:33.915720 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:34.065400 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:34.065356 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:35.818233 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:35.818193 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:35.964358 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:35.964323 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:36.625212 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:36.625173 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:51.928215 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:57:51.928173 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 17:57:55.362763 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:55.362728 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:55.516934 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:55.516899 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:55.653755 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:55.653683 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:59.360666 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:59.360635 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:59.514267 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:57:59.514234 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:01.762973 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:01.762933 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:02.096039 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:02.096006 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:04.875783 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:04.875748 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx"] Oct 30 17:58:04.877910 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:04.877890 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:04.912220 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:04.912194 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx"] Oct 30 17:58:04.983811 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:04.983777 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bbab3a56-2437-46f2-ad50-412736486724-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mddx\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:04.983939 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:04.983817 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh2cn\" (UniqueName: \"kubernetes.io/projected/bbab3a56-2437-46f2-ad50-412736486724-kube-api-access-nh2cn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mddx\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:04.983939 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:04.983841 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbab3a56-2437-46f2-ad50-412736486724-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mddx\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:04.983939 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:04.983909 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbab3a56-2437-46f2-ad50-412736486724-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mddx\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:04.984060 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:04.983960 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbab3a56-2437-46f2-ad50-412736486724-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mddx\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:04.984060 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:04.984013 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bbab3a56-2437-46f2-ad50-412736486724-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mddx\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:04.984060 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:04.984036 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bbab3a56-2437-46f2-ad50-412736486724-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mddx\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:05.085072 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.085039 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bbab3a56-2437-46f2-ad50-412736486724-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mddx\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:05.085203 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.085079 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh2cn\" (UniqueName: \"kubernetes.io/projected/bbab3a56-2437-46f2-ad50-412736486724-kube-api-access-nh2cn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mddx\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:05.085203 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.085106 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbab3a56-2437-46f2-ad50-412736486724-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mddx\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:05.085203 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.085131 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbab3a56-2437-46f2-ad50-412736486724-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mddx\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:05.085203 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.085157 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbab3a56-2437-46f2-ad50-412736486724-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mddx\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:05.085414 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.085210 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bbab3a56-2437-46f2-ad50-412736486724-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mddx\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:05.085414 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.085246 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bbab3a56-2437-46f2-ad50-412736486724-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mddx\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:05.086066 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.085918 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bbab3a56-2437-46f2-ad50-412736486724-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mddx\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:05.087547 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.087522 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbab3a56-2437-46f2-ad50-412736486724-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mddx\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:05.087723 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.087699 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bbab3a56-2437-46f2-ad50-412736486724-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mddx\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:05.087837 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.087814 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbab3a56-2437-46f2-ad50-412736486724-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mddx\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:05.087876 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.087828 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bbab3a56-2437-46f2-ad50-412736486724-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mddx\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:05.108312 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.108284 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbab3a56-2437-46f2-ad50-412736486724-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mddx\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:05.108748 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.108729 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh2cn\" (UniqueName: \"kubernetes.io/projected/bbab3a56-2437-46f2-ad50-412736486724-kube-api-access-nh2cn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mddx\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:05.187539 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.187454 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:05.378055 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.378017 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:05.544860 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.544826 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx"] Oct 30 17:58:05.548568 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:58:05.548541 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbab3a56_2437_46f2_ad50_412736486724.slice/crio-4d83c4c5823ebda01902bd9d9925cfa3b46935db12dd1f8a168d901cd389babe WatchSource:0}: Error finding container 4d83c4c5823ebda01902bd9d9925cfa3b46935db12dd1f8a168d901cd389babe: Status 404 returned error can't find the container with id 4d83c4c5823ebda01902bd9d9925cfa3b46935db12dd1f8a168d901cd389babe Oct 30 17:58:05.551321 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.551294 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:58:05.551389 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.551353 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:58:05.575090 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.575066 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:05.725303 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.725268 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:05.742452 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.742424 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" event={"ID":"bbab3a56-2437-46f2-ad50-412736486724","Type":"ContainerStarted","Data":"01af62d12a42db981979a44b783dbe75b6112844973ad418b19204e4493099a2"} Oct 30 17:58:05.742610 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.742472 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" event={"ID":"bbab3a56-2437-46f2-ad50-412736486724","Type":"ContainerStarted","Data":"4d83c4c5823ebda01902bd9d9925cfa3b46935db12dd1f8a168d901cd389babe"} Oct 30 17:58:05.742610 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.742543 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:05.795234 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.795161 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" podStartSLOduration=1.795150011 podStartE2EDuration="1.795150011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:05.793836185 +0000 UTC m=+346.509406058" watchObservedRunningTime="2026-10-30 17:58:05.795150011 +0000 UTC m=+346.510719884" Oct 30 17:58:05.877745 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:05.877710 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:06.041688 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:06.038689 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:06.217704 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:06.217624 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:06.748524 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:06.748487 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 17:58:06.837684 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:06.837648 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg"] Oct 30 17:58:06.837870 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:06.837848 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" podUID="284f0ec5-9302-475d-95d4-e9a5304ed34c" containerName="discovery" containerID="cri-o://35999d3bbaa647ec206b1486ffc7e7142ef39f1176e52e1d13c41239ce4df1cf" gracePeriod=30 Oct 30 17:58:07.079535 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.079495 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:58:07.204949 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.204925 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-kubeconfig\") pod \"284f0ec5-9302-475d-95d4-e9a5304ed34c\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " Oct 30 17:58:07.205093 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.204981 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-csr-dns-cert\") pod \"284f0ec5-9302-475d-95d4-e9a5304ed34c\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " Oct 30 17:58:07.205093 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.205001 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/284f0ec5-9302-475d-95d4-e9a5304ed34c-cacerts\" (UniqueName: \"kubernetes.io/secret/284f0ec5-9302-475d-95d4-e9a5304ed34c-cacerts\") pod \"284f0ec5-9302-475d-95d4-e9a5304ed34c\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " Oct 30 17:58:07.205175 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.205127 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-token\" (UniqueName: \"kubernetes.io/projected/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-token\") pod \"284f0ec5-9302-475d-95d4-e9a5304ed34c\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " Oct 30 17:58:07.205231 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.205187 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284f0ec5-9302-475d-95d4-e9a5304ed34c-kube-api-access-2pdh2\" (UniqueName: \"kubernetes.io/projected/284f0ec5-9302-475d-95d4-e9a5304ed34c-kube-api-access-2pdh2\") pod \"284f0ec5-9302-475d-95d4-e9a5304ed34c\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " Oct 30 17:58:07.205350 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.205233 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-csr-ca-configmap\") pod \"284f0ec5-9302-475d-95d4-e9a5304ed34c\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " Oct 30 17:58:07.205350 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.205286 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284f0ec5-9302-475d-95d4-e9a5304ed34c-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/284f0ec5-9302-475d-95d4-e9a5304ed34c-local-certs\") pod \"284f0ec5-9302-475d-95d4-e9a5304ed34c\" (UID: \"284f0ec5-9302-475d-95d4-e9a5304ed34c\") " Oct 30 17:58:07.205704 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.205670 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-csr-ca-configmap" pod "284f0ec5-9302-475d-95d4-e9a5304ed34c" (UID: "284f0ec5-9302-475d-95d4-e9a5304ed34c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:58:07.207554 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.207487 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-token" pod "284f0ec5-9302-475d-95d4-e9a5304ed34c" (UID: "284f0ec5-9302-475d-95d4-e9a5304ed34c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:58:07.207643 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.207619 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/284f0ec5-9302-475d-95d4-e9a5304ed34c-cacerts" pod "284f0ec5-9302-475d-95d4-e9a5304ed34c" (UID: "284f0ec5-9302-475d-95d4-e9a5304ed34c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:58:07.207881 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.207861 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284f0ec5-9302-475d-95d4-e9a5304ed34c-kube-api-access-2pdh2" pod "284f0ec5-9302-475d-95d4-e9a5304ed34c" (UID: "284f0ec5-9302-475d-95d4-e9a5304ed34c"). InnerVolumeSpecName "kube-api-access-2pdh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:58:07.207922 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.207866 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-kubeconfig" pod "284f0ec5-9302-475d-95d4-e9a5304ed34c" (UID: "284f0ec5-9302-475d-95d4-e9a5304ed34c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:58:07.208093 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.208075 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-csr-dns-cert" pod "284f0ec5-9302-475d-95d4-e9a5304ed34c" (UID: "284f0ec5-9302-475d-95d4-e9a5304ed34c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:58:07.208093 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.208079 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284f0ec5-9302-475d-95d4-e9a5304ed34c-local-certs" pod "284f0ec5-9302-475d-95d4-e9a5304ed34c" (UID: "284f0ec5-9302-475d-95d4-e9a5304ed34c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:58:07.306322 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.306298 4603 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/284f0ec5-9302-475d-95d4-e9a5304ed34c-local-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:58:07.306322 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.306320 4603 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-kubeconfig\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:58:07.306457 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.306329 4603 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-csr-dns-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:58:07.306457 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.306339 4603 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/284f0ec5-9302-475d-95d4-e9a5304ed34c-cacerts\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:58:07.306457 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.306347 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:58:07.306457 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.306356 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pdh2\" (UniqueName: \"kubernetes.io/projected/284f0ec5-9302-475d-95d4-e9a5304ed34c-kube-api-access-2pdh2\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:58:07.306457 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.306364 4603 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/284f0ec5-9302-475d-95d4-e9a5304ed34c-istio-csr-ca-configmap\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 17:58:07.752248 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.752174 4603 generic.go:356] "Generic (PLEG): container finished" podID="284f0ec5-9302-475d-95d4-e9a5304ed34c" containerID="35999d3bbaa647ec206b1486ffc7e7142ef39f1176e52e1d13c41239ce4df1cf" exitCode=0 Oct 30 17:58:07.752248 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.752224 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" Oct 30 17:58:07.752413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.752257 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" event={"ID":"284f0ec5-9302-475d-95d4-e9a5304ed34c","Type":"ContainerDied","Data":"35999d3bbaa647ec206b1486ffc7e7142ef39f1176e52e1d13c41239ce4df1cf"} Oct 30 17:58:07.752413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.752306 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg" event={"ID":"284f0ec5-9302-475d-95d4-e9a5304ed34c","Type":"ContainerDied","Data":"01a1791979e5880a4aa404dced00cc0b61594a5b07c51a3b1aff7c9d8e0881c3"} Oct 30 17:58:07.752413 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.752326 4603 scope.go:122] "RemoveContainer" containerID="35999d3bbaa647ec206b1486ffc7e7142ef39f1176e52e1d13c41239ce4df1cf" Oct 30 17:58:07.760755 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.760735 4603 scope.go:122] "RemoveContainer" containerID="35999d3bbaa647ec206b1486ffc7e7142ef39f1176e52e1d13c41239ce4df1cf" Oct 30 17:58:07.761031 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:58:07.761010 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35999d3bbaa647ec206b1486ffc7e7142ef39f1176e52e1d13c41239ce4df1cf\": container with ID starting with 35999d3bbaa647ec206b1486ffc7e7142ef39f1176e52e1d13c41239ce4df1cf not found: ID does not exist" containerID="35999d3bbaa647ec206b1486ffc7e7142ef39f1176e52e1d13c41239ce4df1cf" Oct 30 17:58:07.761090 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.761045 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35999d3bbaa647ec206b1486ffc7e7142ef39f1176e52e1d13c41239ce4df1cf"} err="failed to get container status \"35999d3bbaa647ec206b1486ffc7e7142ef39f1176e52e1d13c41239ce4df1cf\": rpc error: code = NotFound desc = could not find container \"35999d3bbaa647ec206b1486ffc7e7142ef39f1176e52e1d13c41239ce4df1cf\": container with ID starting with 35999d3bbaa647ec206b1486ffc7e7142ef39f1176e52e1d13c41239ce4df1cf not found: ID does not exist" Oct 30 17:58:07.813175 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.813153 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg"] Oct 30 17:58:07.825093 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:07.825064 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zkxdg"] Oct 30 17:58:08.057298 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:08.057267 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:08.217371 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:08.217332 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:08.410027 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:08.409952 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:08.558932 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:08.558898 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:09.407468 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:09.407438 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="284f0ec5-9302-475d-95d4-e9a5304ed34c" path="/var/lib/kubelet/pods/284f0ec5-9302-475d-95d4-e9a5304ed34c/volumes" Oct 30 17:58:11.926655 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:11.926615 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:21.352053 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:21.352011 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:21.500248 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:21.500217 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:21.929832 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:58:21.929795 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 17:58:29.231937 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:29.231900 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:29.374563 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:29.374497 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:30.761617 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:30.761579 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:30.908566 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:30.908529 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:31.106243 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:31.106213 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:31.246495 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:31.246464 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:33.440582 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:33.440553 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 17:58:41.126395 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.126351 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:41.278338 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.278307 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:41.597316 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.597285 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-dd6797bcb-qgcnl"] Oct 30 17:58:41.597604 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.597592 4603 cpu_manager.go:414] "containerMap: removing container" podUID="284f0ec5-9302-475d-95d4-e9a5304ed34c" containerName="discovery" Oct 30 17:58:41.597660 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.597605 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="284f0ec5-9302-475d-95d4-e9a5304ed34c" containerName="discovery" Oct 30 17:58:41.597694 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.597678 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="284f0ec5-9302-475d-95d4-e9a5304ed34c" containerName="discovery" Oct 30 17:58:41.600705 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.600690 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd6797bcb-qgcnl" Oct 30 17:58:41.626908 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.626888 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:58:41.626908 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.626887 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 17:58:41.627104 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.626955 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8pmq2\"" Oct 30 17:58:41.627104 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.626967 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:58:41.627104 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.627021 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 17:58:41.628295 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.628275 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd6797bcb-qgcnl"] Oct 30 17:58:41.646580 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.646553 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3c2b36e-0a6b-418d-a91c-b86d4678cb71-metrics-certs\") pod \"maas-api-dd6797bcb-qgcnl\" (UID: \"c3c2b36e-0a6b-418d-a91c-b86d4678cb71\") " pod="odh-ai-gateway-infra/maas-api-dd6797bcb-qgcnl" Oct 30 17:58:41.646674 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.646610 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c2b36e-0a6b-418d-a91c-b86d4678cb71-openshift-service-ca\") pod \"maas-api-dd6797bcb-qgcnl\" (UID: \"c3c2b36e-0a6b-418d-a91c-b86d4678cb71\") " pod="odh-ai-gateway-infra/maas-api-dd6797bcb-qgcnl" Oct 30 17:58:41.646674 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.646669 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnrbf\" (UniqueName: \"kubernetes.io/projected/c3c2b36e-0a6b-418d-a91c-b86d4678cb71-kube-api-access-jnrbf\") pod \"maas-api-dd6797bcb-qgcnl\" (UID: \"c3c2b36e-0a6b-418d-a91c-b86d4678cb71\") " pod="odh-ai-gateway-infra/maas-api-dd6797bcb-qgcnl" Oct 30 17:58:41.646742 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.646708 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3c2b36e-0a6b-418d-a91c-b86d4678cb71-maas-api-tls\") pod \"maas-api-dd6797bcb-qgcnl\" (UID: \"c3c2b36e-0a6b-418d-a91c-b86d4678cb71\") " pod="odh-ai-gateway-infra/maas-api-dd6797bcb-qgcnl" Oct 30 17:58:41.747777 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.747751 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c2b36e-0a6b-418d-a91c-b86d4678cb71-openshift-service-ca\") pod \"maas-api-dd6797bcb-qgcnl\" (UID: \"c3c2b36e-0a6b-418d-a91c-b86d4678cb71\") " pod="odh-ai-gateway-infra/maas-api-dd6797bcb-qgcnl" Oct 30 17:58:41.747907 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.747784 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnrbf\" (UniqueName: \"kubernetes.io/projected/c3c2b36e-0a6b-418d-a91c-b86d4678cb71-kube-api-access-jnrbf\") pod \"maas-api-dd6797bcb-qgcnl\" (UID: \"c3c2b36e-0a6b-418d-a91c-b86d4678cb71\") " pod="odh-ai-gateway-infra/maas-api-dd6797bcb-qgcnl" Oct 30 17:58:41.747907 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.747806 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3c2b36e-0a6b-418d-a91c-b86d4678cb71-maas-api-tls\") pod \"maas-api-dd6797bcb-qgcnl\" (UID: \"c3c2b36e-0a6b-418d-a91c-b86d4678cb71\") " pod="odh-ai-gateway-infra/maas-api-dd6797bcb-qgcnl" Oct 30 17:58:41.747907 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.747864 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3c2b36e-0a6b-418d-a91c-b86d4678cb71-metrics-certs\") pod \"maas-api-dd6797bcb-qgcnl\" (UID: \"c3c2b36e-0a6b-418d-a91c-b86d4678cb71\") " pod="odh-ai-gateway-infra/maas-api-dd6797bcb-qgcnl" Oct 30 17:58:41.748469 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.748447 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c2b36e-0a6b-418d-a91c-b86d4678cb71-openshift-service-ca\") pod \"maas-api-dd6797bcb-qgcnl\" (UID: \"c3c2b36e-0a6b-418d-a91c-b86d4678cb71\") " pod="odh-ai-gateway-infra/maas-api-dd6797bcb-qgcnl" Oct 30 17:58:41.750401 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.750376 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3c2b36e-0a6b-418d-a91c-b86d4678cb71-metrics-certs\") pod \"maas-api-dd6797bcb-qgcnl\" (UID: \"c3c2b36e-0a6b-418d-a91c-b86d4678cb71\") " pod="odh-ai-gateway-infra/maas-api-dd6797bcb-qgcnl" Oct 30 17:58:41.750564 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.750546 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3c2b36e-0a6b-418d-a91c-b86d4678cb71-maas-api-tls\") pod \"maas-api-dd6797bcb-qgcnl\" (UID: \"c3c2b36e-0a6b-418d-a91c-b86d4678cb71\") " pod="odh-ai-gateway-infra/maas-api-dd6797bcb-qgcnl" Oct 30 17:58:41.770643 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.770620 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnrbf\" (UniqueName: \"kubernetes.io/projected/c3c2b36e-0a6b-418d-a91c-b86d4678cb71-kube-api-access-jnrbf\") pod \"maas-api-dd6797bcb-qgcnl\" (UID: \"c3c2b36e-0a6b-418d-a91c-b86d4678cb71\") " pod="odh-ai-gateway-infra/maas-api-dd6797bcb-qgcnl" Oct 30 17:58:41.913421 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:41.913364 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd6797bcb-qgcnl" Oct 30 17:58:42.213581 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:42.213490 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd6797bcb-qgcnl"] Oct 30 17:58:42.216689 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:58:42.216656 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c2b36e_0a6b_418d_a91c_b86d4678cb71.slice/crio-bfb515b7d4d4c7700f70f82b86d1f76f2c5ac41375741157c71a9f1e5db8269c WatchSource:0}: Error finding container bfb515b7d4d4c7700f70f82b86d1f76f2c5ac41375741157c71a9f1e5db8269c: Status 404 returned error can't find the container with id bfb515b7d4d4c7700f70f82b86d1f76f2c5ac41375741157c71a9f1e5db8269c Oct 30 17:58:42.217883 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:42.217866 4603 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:58:42.890992 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:42.890951 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd6797bcb-qgcnl" event={"ID":"c3c2b36e-0a6b-418d-a91c-b86d4678cb71","Type":"ContainerStarted","Data":"bfb515b7d4d4c7700f70f82b86d1f76f2c5ac41375741157c71a9f1e5db8269c"} Oct 30 17:58:44.901716 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:44.901682 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd6797bcb-qgcnl" event={"ID":"c3c2b36e-0a6b-418d-a91c-b86d4678cb71","Type":"ContainerStarted","Data":"5e9b829159518d8858ffb4a3aee8f7417fdf9e3398923332336fe988d6fc5caf"} Oct 30 17:58:44.902135 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:44.901783 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-dd6797bcb-qgcnl" Oct 30 17:58:44.942733 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:44.942683 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-dd6797bcb-qgcnl" podStartSLOduration=1.754020667 podStartE2EDuration="3.942672004s" totalImagesPullingTime="2.188651337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:44.941822451 +0000 UTC m=+385.657392325" watchObservedRunningTime="2026-10-30 17:58:44.942672004 +0000 UTC m=+385.658241901" Oct 30 17:58:50.912228 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:50.912197 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-dd6797bcb-qgcnl" Oct 30 17:58:51.927788 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:58:51.927755 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 17:58:59.413046 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:59.413005 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:59.553883 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:58:59.553850 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:09.494830 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:09.494745 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:09.640356 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:09.640314 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:12.033903 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:12.033868 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:21.928476 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:59:21.928437 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 17:59:38.091480 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.091448 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr"] Oct 30 17:59:38.094706 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.094687 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.111625 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.111606 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kj7cx\"" Oct 30 17:59:38.124453 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.124428 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr"] Oct 30 17:59:38.251430 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.251400 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.251602 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.251437 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.251602 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.251466 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.251602 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.251594 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.251712 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.251623 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.251712 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.251642 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjwz8\" (UniqueName: \"kubernetes.io/projected/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-kube-api-access-hjwz8\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.251798 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.251776 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.251833 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.251823 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.251877 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.251865 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.353087 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.353019 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjwz8\" (UniqueName: \"kubernetes.io/projected/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-kube-api-access-hjwz8\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.353087 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.353081 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.353296 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.353108 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.353296 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.353126 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.353296 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.353150 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.353296 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.353166 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.353296 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.353184 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.353622 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.353332 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.353622 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.353378 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.353622 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.353560 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.353736 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.353613 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.353850 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.353830 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.353948 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.353925 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.354005 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.353960 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.355437 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.355415 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.355910 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.355895 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.378052 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.378029 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjwz8\" (UniqueName: \"kubernetes.io/projected/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-kube-api-access-hjwz8\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.378155 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.378085 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-fhwtr\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.406932 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.406914 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:38.720523 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:38.720475 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr"] Oct 30 17:59:38.723175 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:59:38.723147 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049906ec_ccdb_45e9_8fe9_9f50150fc7f6.slice/crio-de2dc335016ad12f1d82771f8508728b6483fdf340ff24fc1e1c212adadd7d37 WatchSource:0}: Error finding container de2dc335016ad12f1d82771f8508728b6483fdf340ff24fc1e1c212adadd7d37: Status 404 returned error can't find the container with id de2dc335016ad12f1d82771f8508728b6483fdf340ff24fc1e1c212adadd7d37 Oct 30 17:59:39.119521 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:39.119465 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" event={"ID":"049906ec-ccdb-45e9-8fe9-9f50150fc7f6","Type":"ContainerStarted","Data":"de2dc335016ad12f1d82771f8508728b6483fdf340ff24fc1e1c212adadd7d37"} Oct 30 17:59:40.227287 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:40.227249 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:40.368996 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:40.368962 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:41.215945 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:41.215915 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:59:41.216063 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:41.215981 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:59:41.216063 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:41.216011 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:59:42.134081 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:42.134049 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" event={"ID":"049906ec-ccdb-45e9-8fe9-9f50150fc7f6","Type":"ContainerStarted","Data":"f18a4dbda2fc46a13c70f70884ce40718e12029125e3b680bd57fa84f651a413"} Oct 30 17:59:42.214082 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:42.214021 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" podStartSLOduration=1.723818437 podStartE2EDuration="4.214005715s" totalImagesPullingTime="2.490187278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:42.197958475 +0000 UTC m=+442.913528380" watchObservedRunningTime="2026-10-30 17:59:42.214005715 +0000 UTC m=+442.929575588" Oct 30 17:59:42.413579 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:42.410627 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:42.413579 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:42.411330 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fhwtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 30 17:59:42.413579 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:42.411378 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" podUID="049906ec-ccdb-45e9-8fe9-9f50150fc7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 30 17:59:43.413668 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:43.413635 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fhwtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 30 17:59:43.414185 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:43.413694 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" podUID="049906ec-ccdb-45e9-8fe9-9f50150fc7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 30 17:59:44.015038 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:44.015007 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:44.189886 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:44.189853 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:44.413530 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:44.410677 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fhwtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 30 17:59:44.413530 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:44.410729 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" podUID="049906ec-ccdb-45e9-8fe9-9f50150fc7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 30 17:59:45.408912 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:45.408874 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fhwtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 30 17:59:45.409307 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:45.409273 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" podUID="049906ec-ccdb-45e9-8fe9-9f50150fc7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 30 17:59:46.410538 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:46.409882 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fhwtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 30 17:59:46.410538 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:46.409953 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" podUID="049906ec-ccdb-45e9-8fe9-9f50150fc7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 30 17:59:47.518982 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:47.518941 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:47.519487 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:47.519466 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:47.520955 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:47.520935 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 17:59:51.928752 ip-10-0-128-236 kubenswrapper[4603]: E1030 17:59:51.928715 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 17:59:53.118673 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.118635 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh"] Oct 30 17:59:53.122206 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.122163 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.153609 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.153584 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh"] Oct 30 17:59:53.296620 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.296589 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53429703-95ae-4f02-8c26-a9a296c56a05-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.296620 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.296620 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53429703-95ae-4f02-8c26-a9a296c56a05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.296872 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.296655 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53429703-95ae-4f02-8c26-a9a296c56a05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.296872 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.296714 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8dpn\" (UniqueName: \"kubernetes.io/projected/53429703-95ae-4f02-8c26-a9a296c56a05-kube-api-access-d8dpn\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.296872 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.296807 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.296872 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.296842 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.296872 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.296860 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.297103 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.296900 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.297103 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.296942 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.397373 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.397306 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53429703-95ae-4f02-8c26-a9a296c56a05-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.397373 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.397332 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53429703-95ae-4f02-8c26-a9a296c56a05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.397373 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.397362 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53429703-95ae-4f02-8c26-a9a296c56a05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.397636 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.397561 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8dpn\" (UniqueName: \"kubernetes.io/projected/53429703-95ae-4f02-8c26-a9a296c56a05-kube-api-access-d8dpn\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.397636 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.397618 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.397759 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.397654 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.397825 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.397801 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.397878 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.397849 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.398053 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.398031 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.398234 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.398209 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.398518 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.398185 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53429703-95ae-4f02-8c26-a9a296c56a05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.398518 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.398041 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.398518 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.398343 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.398518 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.398341 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.399924 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.399903 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53429703-95ae-4f02-8c26-a9a296c56a05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.400047 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.400026 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.420176 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.420157 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53429703-95ae-4f02-8c26-a9a296c56a05-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.420380 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.420359 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8dpn\" (UniqueName: \"kubernetes.io/projected/53429703-95ae-4f02-8c26-a9a296c56a05-kube-api-access-d8dpn\") pod \"maas-default-gateway-openshift-default-67bf986f99-qmzkh\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.434124 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.434103 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:53.749224 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.749185 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh"] Oct 30 17:59:53.751754 ip-10-0-128-236 kubenswrapper[4603]: W1030 17:59:53.751725 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53429703_95ae_4f02_8c26_a9a296c56a05.slice/crio-cfb7f8588d3dd9cc11e92d7f17b43c471d7e6d8b8686a6c83b57bcbb87a8214b WatchSource:0}: Error finding container cfb7f8588d3dd9cc11e92d7f17b43c471d7e6d8b8686a6c83b57bcbb87a8214b: Status 404 returned error can't find the container with id cfb7f8588d3dd9cc11e92d7f17b43c471d7e6d8b8686a6c83b57bcbb87a8214b Oct 30 17:59:53.753873 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.753841 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:59:53.753967 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.753902 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:59:53.753967 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:53.753928 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:59:54.215441 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:54.215403 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" event={"ID":"53429703-95ae-4f02-8c26-a9a296c56a05","Type":"ContainerStarted","Data":"7e2406130b23b97ceafe24ac4af504bcc85fe97a8a0703755fd7c01cbadefa7e"} Oct 30 17:59:54.215928 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:54.215460 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" event={"ID":"53429703-95ae-4f02-8c26-a9a296c56a05","Type":"ContainerStarted","Data":"cfb7f8588d3dd9cc11e92d7f17b43c471d7e6d8b8686a6c83b57bcbb87a8214b"} Oct 30 17:59:54.282437 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:54.282386 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" podStartSLOduration=1.282373629 podStartE2EDuration="1.282373629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:54.269918753 +0000 UTC m=+454.985488626" watchObservedRunningTime="2026-10-30 17:59:54.282373629 +0000 UTC m=+454.997943502" Oct 30 17:59:54.434979 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:54.434947 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 17:59:54.436678 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:54.436639 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qmzkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 17:59:54.436812 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:54.436708 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" podUID="53429703-95ae-4f02-8c26-a9a296c56a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 17:59:55.435274 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:55.435240 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qmzkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 17:59:55.435705 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:55.435294 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" podUID="53429703-95ae-4f02-8c26-a9a296c56a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 17:59:56.434788 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:56.434759 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qmzkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 17:59:56.434954 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:56.434829 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" podUID="53429703-95ae-4f02-8c26-a9a296c56a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 17:59:57.435307 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:57.435276 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qmzkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 17:59:57.435728 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:57.435323 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" podUID="53429703-95ae-4f02-8c26-a9a296c56a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 17:59:58.434986 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:58.434952 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qmzkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 17:59:58.435160 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:58.435007 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" podUID="53429703-95ae-4f02-8c26-a9a296c56a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 17:59:59.435310 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:59.435277 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qmzkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 17:59:59.435698 ip-10-0-128-236 kubenswrapper[4603]: I1030 17:59:59.435350 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" podUID="53429703-95ae-4f02-8c26-a9a296c56a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 18:00:00.439682 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:00.439655 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 18:00:00.440051 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:00.439945 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 18:00:00.440601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:00.440586 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 18:00:12.166273 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:12.166236 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:21.929315 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:00:21.929278 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:00:28.882238 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:28.882197 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:29.424264 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:29.424219 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:40.764088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:40.764010 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:40.908925 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:40.908893 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:49.764089 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:49.764040 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:51.183992 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:51.183957 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:00:51.929958 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:00:51.929921 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:00:54.966467 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:54.966436 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv"] Oct 30 18:00:54.970391 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:54.970368 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:00:55.015659 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.015636 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 18:00:55.015833 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.015666 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9lnst\"" Oct 30 18:00:55.015944 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.015757 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 18:00:55.016201 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.016180 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 18:00:55.017248 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.017227 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv"] Oct 30 18:00:55.054032 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.054009 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4343775-794f-4a3c-8732-01eeff138bbe-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qhvxv\" (UID: \"f4343775-794f-4a3c-8732-01eeff138bbe\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:00:55.054163 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.054083 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nvm7\" (UniqueName: \"kubernetes.io/projected/f4343775-794f-4a3c-8732-01eeff138bbe-kube-api-access-4nvm7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qhvxv\" (UID: \"f4343775-794f-4a3c-8732-01eeff138bbe\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:00:55.054163 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.054128 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4343775-794f-4a3c-8732-01eeff138bbe-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qhvxv\" (UID: \"f4343775-794f-4a3c-8732-01eeff138bbe\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:00:55.054163 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.054159 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4343775-794f-4a3c-8732-01eeff138bbe-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qhvxv\" (UID: \"f4343775-794f-4a3c-8732-01eeff138bbe\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:00:55.054298 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.054177 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4343775-794f-4a3c-8732-01eeff138bbe-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qhvxv\" (UID: \"f4343775-794f-4a3c-8732-01eeff138bbe\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:00:55.054298 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.054266 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4343775-794f-4a3c-8732-01eeff138bbe-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qhvxv\" (UID: \"f4343775-794f-4a3c-8732-01eeff138bbe\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:00:55.153325 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.153291 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:55.155392 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.155367 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4343775-794f-4a3c-8732-01eeff138bbe-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qhvxv\" (UID: \"f4343775-794f-4a3c-8732-01eeff138bbe\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:00:55.155529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.155412 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nvm7\" (UniqueName: \"kubernetes.io/projected/f4343775-794f-4a3c-8732-01eeff138bbe-kube-api-access-4nvm7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qhvxv\" (UID: \"f4343775-794f-4a3c-8732-01eeff138bbe\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:00:55.155529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.155450 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4343775-794f-4a3c-8732-01eeff138bbe-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qhvxv\" (UID: \"f4343775-794f-4a3c-8732-01eeff138bbe\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:00:55.155529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.155480 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4343775-794f-4a3c-8732-01eeff138bbe-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qhvxv\" (UID: \"f4343775-794f-4a3c-8732-01eeff138bbe\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:00:55.155529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.155517 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4343775-794f-4a3c-8732-01eeff138bbe-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qhvxv\" (UID: \"f4343775-794f-4a3c-8732-01eeff138bbe\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:00:55.155710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.155563 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4343775-794f-4a3c-8732-01eeff138bbe-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qhvxv\" (UID: \"f4343775-794f-4a3c-8732-01eeff138bbe\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:00:55.155970 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.155937 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4343775-794f-4a3c-8732-01eeff138bbe-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qhvxv\" (UID: \"f4343775-794f-4a3c-8732-01eeff138bbe\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:00:55.156076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.156012 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4343775-794f-4a3c-8732-01eeff138bbe-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qhvxv\" (UID: \"f4343775-794f-4a3c-8732-01eeff138bbe\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:00:55.156076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.156014 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4343775-794f-4a3c-8732-01eeff138bbe-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qhvxv\" (UID: \"f4343775-794f-4a3c-8732-01eeff138bbe\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:00:55.157805 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.157789 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4343775-794f-4a3c-8732-01eeff138bbe-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qhvxv\" (UID: \"f4343775-794f-4a3c-8732-01eeff138bbe\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:00:55.158141 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.158123 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4343775-794f-4a3c-8732-01eeff138bbe-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qhvxv\" (UID: \"f4343775-794f-4a3c-8732-01eeff138bbe\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:00:55.178258 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.178235 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nvm7\" (UniqueName: \"kubernetes.io/projected/f4343775-794f-4a3c-8732-01eeff138bbe-kube-api-access-4nvm7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qhvxv\" (UID: \"f4343775-794f-4a3c-8732-01eeff138bbe\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:00:55.286490 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.286465 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:00:55.586705 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:55.586680 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv"] Oct 30 18:00:55.589487 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:00:55.589461 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4343775_794f_4a3c_8732_01eeff138bbe.slice/crio-d321ca71374f39974ce4265e7f71c2b2e89bb4695607746a841516c9d3d59fae WatchSource:0}: Error finding container d321ca71374f39974ce4265e7f71c2b2e89bb4695607746a841516c9d3d59fae: Status 404 returned error can't find the container with id d321ca71374f39974ce4265e7f71c2b2e89bb4695607746a841516c9d3d59fae Oct 30 18:00:56.473883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:56.473846 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" event={"ID":"f4343775-794f-4a3c-8732-01eeff138bbe","Type":"ContainerStarted","Data":"d321ca71374f39974ce4265e7f71c2b2e89bb4695607746a841516c9d3d59fae"} Oct 30 18:00:57.012948 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:57.012909 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:00:57.227321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:57.227272 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:57.479051 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:57.479014 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" event={"ID":"f4343775-794f-4a3c-8732-01eeff138bbe","Type":"ContainerStarted","Data":"3198a9a421e2ddb46ce7c77f2155e0eebcc34b672486af2bc97a0ca0443528c9"} Oct 30 18:00:57.520227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:57.520179 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" podStartSLOduration=2.445142023 podStartE2EDuration="3.520161382s" totalImagesPullingTime="1.075019359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:57.519111573 +0000 UTC m=+518.234681448" watchObservedRunningTime="2026-10-30 18:00:57.520161382 +0000 UTC m=+518.235731256" Oct 30 18:00:58.287306 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.287274 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:00:58.775704 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.775674 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx"] Oct 30 18:00:58.778965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.778941 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:00:58.794257 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.794231 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 18:00:58.806846 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.806825 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx"] Oct 30 18:00:58.869142 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.869114 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:58.895143 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.895119 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1112c14a-18f4-46de-ba0c-8e28f19ef80d-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zz7vx\" (UID: \"1112c14a-18f4-46de-ba0c-8e28f19ef80d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:00:58.895229 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.895173 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1112c14a-18f4-46de-ba0c-8e28f19ef80d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zz7vx\" (UID: \"1112c14a-18f4-46de-ba0c-8e28f19ef80d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:00:58.895282 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.895268 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1112c14a-18f4-46de-ba0c-8e28f19ef80d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zz7vx\" (UID: \"1112c14a-18f4-46de-ba0c-8e28f19ef80d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:00:58.895347 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.895292 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f6mg\" (UniqueName: \"kubernetes.io/projected/1112c14a-18f4-46de-ba0c-8e28f19ef80d-kube-api-access-9f6mg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zz7vx\" (UID: \"1112c14a-18f4-46de-ba0c-8e28f19ef80d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:00:58.895387 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.895329 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1112c14a-18f4-46de-ba0c-8e28f19ef80d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zz7vx\" (UID: \"1112c14a-18f4-46de-ba0c-8e28f19ef80d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:00:58.895450 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.895432 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1112c14a-18f4-46de-ba0c-8e28f19ef80d-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zz7vx\" (UID: \"1112c14a-18f4-46de-ba0c-8e28f19ef80d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:00:58.912731 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.912706 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:00:58.996829 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.996797 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1112c14a-18f4-46de-ba0c-8e28f19ef80d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zz7vx\" (UID: \"1112c14a-18f4-46de-ba0c-8e28f19ef80d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:00:58.997007 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.996848 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1112c14a-18f4-46de-ba0c-8e28f19ef80d-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zz7vx\" (UID: \"1112c14a-18f4-46de-ba0c-8e28f19ef80d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:00:58.997007 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.996871 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1112c14a-18f4-46de-ba0c-8e28f19ef80d-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zz7vx\" (UID: \"1112c14a-18f4-46de-ba0c-8e28f19ef80d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:00:58.997007 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.996946 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1112c14a-18f4-46de-ba0c-8e28f19ef80d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zz7vx\" (UID: \"1112c14a-18f4-46de-ba0c-8e28f19ef80d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:00:58.997007 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.996980 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1112c14a-18f4-46de-ba0c-8e28f19ef80d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zz7vx\" (UID: \"1112c14a-18f4-46de-ba0c-8e28f19ef80d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:00:58.997007 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.997004 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f6mg\" (UniqueName: \"kubernetes.io/projected/1112c14a-18f4-46de-ba0c-8e28f19ef80d-kube-api-access-9f6mg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zz7vx\" (UID: \"1112c14a-18f4-46de-ba0c-8e28f19ef80d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:00:58.997339 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.997309 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1112c14a-18f4-46de-ba0c-8e28f19ef80d-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zz7vx\" (UID: \"1112c14a-18f4-46de-ba0c-8e28f19ef80d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:00:58.997488 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.997389 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1112c14a-18f4-46de-ba0c-8e28f19ef80d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zz7vx\" (UID: \"1112c14a-18f4-46de-ba0c-8e28f19ef80d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:00:58.997488 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.997407 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1112c14a-18f4-46de-ba0c-8e28f19ef80d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zz7vx\" (UID: \"1112c14a-18f4-46de-ba0c-8e28f19ef80d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:00:58.999342 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.999320 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1112c14a-18f4-46de-ba0c-8e28f19ef80d-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zz7vx\" (UID: \"1112c14a-18f4-46de-ba0c-8e28f19ef80d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:00:58.999664 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:58.999647 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1112c14a-18f4-46de-ba0c-8e28f19ef80d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zz7vx\" (UID: \"1112c14a-18f4-46de-ba0c-8e28f19ef80d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:00:59.021788 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:59.021767 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f6mg\" (UniqueName: \"kubernetes.io/projected/1112c14a-18f4-46de-ba0c-8e28f19ef80d-kube-api-access-9f6mg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zz7vx\" (UID: \"1112c14a-18f4-46de-ba0c-8e28f19ef80d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:00:59.091786 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:59.091740 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:00:59.390955 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:59.390910 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx"] Oct 30 18:00:59.393633 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:00:59.393609 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1112c14a_18f4_46de_ba0c_8e28f19ef80d.slice/crio-42a731c602e2491a811ba4d3f5aa0c616d34eb80b24c236d8af3bbc30687f21d WatchSource:0}: Error finding container 42a731c602e2491a811ba4d3f5aa0c616d34eb80b24c236d8af3bbc30687f21d: Status 404 returned error can't find the container with id 42a731c602e2491a811ba4d3f5aa0c616d34eb80b24c236d8af3bbc30687f21d Oct 30 18:00:59.490259 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:00:59.490223 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" event={"ID":"1112c14a-18f4-46de-ba0c-8e28f19ef80d","Type":"ContainerStarted","Data":"42a731c602e2491a811ba4d3f5aa0c616d34eb80b24c236d8af3bbc30687f21d"} Oct 30 18:01:00.498454 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:00.498343 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" event={"ID":"1112c14a-18f4-46de-ba0c-8e28f19ef80d","Type":"ContainerStarted","Data":"1fb8ee8fcbefbd4127e30eb4fbae10b2db57f70ff5d47df3f7a3337a53abec14"} Oct 30 18:01:00.546869 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:00.546796 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" podStartSLOduration=1.726436595 podStartE2EDuration="2.546777584s" totalImagesPullingTime="820.340989ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:00.546561365 +0000 UTC m=+521.262131240" watchObservedRunningTime="2026-10-30 18:01:00.546777584 +0000 UTC m=+521.262347459" Oct 30 18:01:00.732234 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:00.732199 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:01.092447 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:01.092417 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:01:01.529703 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:01.529679 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:01:02.597950 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.597921 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-7mmmn"] Oct 30 18:01:02.601269 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.601250 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:02.626853 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.626828 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 18:01:02.629586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.629564 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-7mmmn"] Oct 30 18:01:02.676551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.676525 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:02.736733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.736707 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0702ace-9c54-4015-add0-e25ee9feaf40-home\") pod \"echo-simulator-kserve-64b6f8f96c-7mmmn\" (UID: \"f0702ace-9c54-4015-add0-e25ee9feaf40\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:02.736854 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.736737 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdjql\" (UniqueName: \"kubernetes.io/projected/f0702ace-9c54-4015-add0-e25ee9feaf40-kube-api-access-bdjql\") pod \"echo-simulator-kserve-64b6f8f96c-7mmmn\" (UID: \"f0702ace-9c54-4015-add0-e25ee9feaf40\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:02.736854 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.736769 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0702ace-9c54-4015-add0-e25ee9feaf40-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-7mmmn\" (UID: \"f0702ace-9c54-4015-add0-e25ee9feaf40\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:02.736952 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.736855 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0702ace-9c54-4015-add0-e25ee9feaf40-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-7mmmn\" (UID: \"f0702ace-9c54-4015-add0-e25ee9feaf40\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:02.736952 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.736873 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0702ace-9c54-4015-add0-e25ee9feaf40-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-7mmmn\" (UID: \"f0702ace-9c54-4015-add0-e25ee9feaf40\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:02.736952 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.736889 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0702ace-9c54-4015-add0-e25ee9feaf40-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-7mmmn\" (UID: \"f0702ace-9c54-4015-add0-e25ee9feaf40\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:02.837625 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.837588 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0702ace-9c54-4015-add0-e25ee9feaf40-home\") pod \"echo-simulator-kserve-64b6f8f96c-7mmmn\" (UID: \"f0702ace-9c54-4015-add0-e25ee9feaf40\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:02.837625 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.837628 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdjql\" (UniqueName: \"kubernetes.io/projected/f0702ace-9c54-4015-add0-e25ee9feaf40-kube-api-access-bdjql\") pod \"echo-simulator-kserve-64b6f8f96c-7mmmn\" (UID: \"f0702ace-9c54-4015-add0-e25ee9feaf40\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:02.837852 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.837681 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0702ace-9c54-4015-add0-e25ee9feaf40-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-7mmmn\" (UID: \"f0702ace-9c54-4015-add0-e25ee9feaf40\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:02.837852 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.837732 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0702ace-9c54-4015-add0-e25ee9feaf40-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-7mmmn\" (UID: \"f0702ace-9c54-4015-add0-e25ee9feaf40\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:02.837852 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.837761 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0702ace-9c54-4015-add0-e25ee9feaf40-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-7mmmn\" (UID: \"f0702ace-9c54-4015-add0-e25ee9feaf40\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:02.837852 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.837788 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0702ace-9c54-4015-add0-e25ee9feaf40-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-7mmmn\" (UID: \"f0702ace-9c54-4015-add0-e25ee9feaf40\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:02.838177 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.838150 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0702ace-9c54-4015-add0-e25ee9feaf40-home\") pod \"echo-simulator-kserve-64b6f8f96c-7mmmn\" (UID: \"f0702ace-9c54-4015-add0-e25ee9feaf40\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:02.838285 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.838183 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0702ace-9c54-4015-add0-e25ee9feaf40-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-7mmmn\" (UID: \"f0702ace-9c54-4015-add0-e25ee9feaf40\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:02.838529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.838470 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0702ace-9c54-4015-add0-e25ee9feaf40-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-7mmmn\" (UID: \"f0702ace-9c54-4015-add0-e25ee9feaf40\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:02.840280 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.840252 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0702ace-9c54-4015-add0-e25ee9feaf40-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-7mmmn\" (UID: \"f0702ace-9c54-4015-add0-e25ee9feaf40\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:02.840658 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.840639 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0702ace-9c54-4015-add0-e25ee9feaf40-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-7mmmn\" (UID: \"f0702ace-9c54-4015-add0-e25ee9feaf40\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:02.860621 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.860567 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdjql\" (UniqueName: \"kubernetes.io/projected/f0702ace-9c54-4015-add0-e25ee9feaf40-kube-api-access-bdjql\") pod \"echo-simulator-kserve-64b6f8f96c-7mmmn\" (UID: \"f0702ace-9c54-4015-add0-e25ee9feaf40\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:02.915343 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:02.915319 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:03.225830 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:03.225054 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-7mmmn"] Oct 30 18:01:03.235145 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:01:03.235106 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0702ace_9c54_4015_add0_e25ee9feaf40.slice/crio-f76a13af1e6134163a1a57c198ecaf80686a5e5744e641523a9e29726f75e696 WatchSource:0}: Error finding container f76a13af1e6134163a1a57c198ecaf80686a5e5744e641523a9e29726f75e696: Status 404 returned error can't find the container with id f76a13af1e6134163a1a57c198ecaf80686a5e5744e641523a9e29726f75e696 Oct 30 18:01:03.527615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:03.527574 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" event={"ID":"f0702ace-9c54-4015-add0-e25ee9feaf40","Type":"ContainerStarted","Data":"f76a13af1e6134163a1a57c198ecaf80686a5e5744e641523a9e29726f75e696"} Oct 30 18:01:04.024238 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:04.024102 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:05.013352 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:05.013311 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:01:05.287686 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:05.287658 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:01:05.299616 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:05.299593 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:01:05.537096 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:05.537059 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" event={"ID":"f0702ace-9c54-4015-add0-e25ee9feaf40","Type":"ContainerStarted","Data":"40446326a0c11ebb6984850850e654b4094cfc57f9db0dfc3cbe6e3ddfc5149a"} Oct 30 18:01:05.548341 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:05.548275 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" Oct 30 18:01:05.582997 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:05.582955 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" podStartSLOduration=1.708769355 podStartE2EDuration="3.582943558s" totalImagesPullingTime="1.874174203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:05.582039191 +0000 UTC m=+526.297609065" watchObservedRunningTime="2026-10-30 18:01:05.582943558 +0000 UTC m=+526.298513469" Oct 30 18:01:05.916359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:05.916282 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:06.754552 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:06.754519 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf"] Oct 30 18:01:06.758058 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:06.758032 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:06.772313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:06.772287 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 18:01:06.784919 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:06.784895 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf"] Oct 30 18:01:06.880395 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:06.880366 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf\" (UID: \"ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:06.880579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:06.880454 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf\" (UID: \"ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:06.880579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:06.880489 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf\" (UID: \"ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:06.880579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:06.880526 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf\" (UID: \"ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:06.880762 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:06.880630 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf\" (UID: \"ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:06.880762 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:06.880668 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9gkx\" (UniqueName: \"kubernetes.io/projected/ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce-kube-api-access-l9gkx\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf\" (UID: \"ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:06.944304 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:06.944276 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:06.981322 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:06.981294 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf\" (UID: \"ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:06.981521 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:06.981339 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf\" (UID: \"ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:06.981521 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:06.981386 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf\" (UID: \"ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:06.981521 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:06.981412 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf\" (UID: \"ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:06.981521 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:06.981444 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf\" (UID: \"ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:06.981521 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:06.981479 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9gkx\" (UniqueName: \"kubernetes.io/projected/ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce-kube-api-access-l9gkx\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf\" (UID: \"ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:06.982032 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:06.982006 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf\" (UID: \"ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:06.982143 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:06.982027 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf\" (UID: \"ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:06.982201 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:06.982175 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf\" (UID: \"ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:06.984106 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:06.984079 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf\" (UID: \"ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:06.984688 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:06.984665 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf\" (UID: \"ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:07.005174 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:07.005119 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9gkx\" (UniqueName: \"kubernetes.io/projected/ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce-kube-api-access-l9gkx\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf\" (UID: \"ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:07.075861 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:07.075836 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:07.382841 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:07.382810 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf"] Oct 30 18:01:07.385877 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:01:07.385847 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba9c2eff_9e64_4f3f_ba4c_a6339d0afbce.slice/crio-09712df840021c90d6ea8f91d49b898187cf7de70d0000e85c51f0223eb756f8 WatchSource:0}: Error finding container 09712df840021c90d6ea8f91d49b898187cf7de70d0000e85c51f0223eb756f8: Status 404 returned error can't find the container with id 09712df840021c90d6ea8f91d49b898187cf7de70d0000e85c51f0223eb756f8 Oct 30 18:01:07.550943 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:07.550882 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" event={"ID":"ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce","Type":"ContainerStarted","Data":"09712df840021c90d6ea8f91d49b898187cf7de70d0000e85c51f0223eb756f8"} Oct 30 18:01:08.373450 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.373421 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522"] Oct 30 18:01:08.376893 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.376873 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.413525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.412943 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn"] Oct 30 18:01:08.418186 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.416731 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.450855 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.450834 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522"] Oct 30 18:01:08.452110 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.452090 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn"] Oct 30 18:01:08.518093 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.518068 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.518878 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.518860 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j4mk\" (UniqueName: \"kubernetes.io/projected/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-kube-api-access-8j4mk\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.519095 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.519050 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.519209 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.519199 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.519332 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.519322 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.519447 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.519437 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.519839 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.519582 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.519984 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.519973 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.520112 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.520101 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.520220 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.520210 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bf7d070-07da-4ad5-8464-e39192f8e23e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.520349 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.520332 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.520447 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.520436 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cxhw\" (UniqueName: \"kubernetes.io/projected/5bf7d070-07da-4ad5-8464-e39192f8e23e-kube-api-access-7cxhw\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.520844 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.520583 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.520985 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.520973 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.521100 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.521090 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.521240 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.521214 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.521350 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.521340 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.521480 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.521471 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.556962 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.556933 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" event={"ID":"ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce","Type":"ContainerStarted","Data":"9b8a6e1862f62c7850d83b441c55d41e8013bf9af2afa1390035410392ca48f3"} Oct 30 18:01:08.598639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.598593 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podStartSLOduration=2.364057449 podStartE2EDuration="2.598581707s" totalImagesPullingTime="234.524258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:08.597244946 +0000 UTC m=+529.312814821" watchObservedRunningTime="2026-10-30 18:01:08.598581707 +0000 UTC m=+529.314151581" Oct 30 18:01:08.622359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.622337 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.622519 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.622364 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.622519 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.622380 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j4mk\" (UniqueName: \"kubernetes.io/projected/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-kube-api-access-8j4mk\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.622519 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.622397 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.622519 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.622417 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.622519 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.622444 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.622519 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.622471 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.622867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.622526 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.622867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.622555 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.622867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.622579 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.622867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.622606 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bf7d070-07da-4ad5-8464-e39192f8e23e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.622867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.622662 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.622867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.622688 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cxhw\" (UniqueName: \"kubernetes.io/projected/5bf7d070-07da-4ad5-8464-e39192f8e23e-kube-api-access-7cxhw\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.622867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.622718 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.622867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.622761 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.622867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.622788 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.622867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.622846 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.623407 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.622861 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.623407 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.622876 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.623407 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.622965 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.623407 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.623046 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.623407 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.623168 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.623733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.623451 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.623733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.623562 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.623733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.623636 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.623733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.623716 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.623958 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.623803 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.624246 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.624200 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bf7d070-07da-4ad5-8464-e39192f8e23e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.625892 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.625866 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.626262 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.626244 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.626359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.626304 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.627024 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.627006 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.645801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.645780 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.645945 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.645872 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cxhw\" (UniqueName: \"kubernetes.io/projected/5bf7d070-07da-4ad5-8464-e39192f8e23e-kube-api-access-7cxhw\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.646275 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.646252 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-x4522\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.646452 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.646434 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j4mk\" (UniqueName: \"kubernetes.io/projected/c77c24c9-98c1-43c8-9460-e5b3c509c6d3-kube-api-access-8j4mk\") pod \"maas-default-gateway-openshift-default-67bf986f99-wjstn\" (UID: \"c77c24c9-98c1-43c8-9460-e5b3c509c6d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:08.689992 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.689961 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:08.739576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:08.739549 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:09.038071 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:09.038042 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522"] Oct 30 18:01:09.046528 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:01:09.046481 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf7d070_07da_4ad5_8464_e39192f8e23e.slice/crio-b67cf4b1f3224c667801624ea198242013f3164d33bd8a7362a3e4cd0ff77720 WatchSource:0}: Error finding container b67cf4b1f3224c667801624ea198242013f3164d33bd8a7362a3e4cd0ff77720: Status 404 returned error can't find the container with id b67cf4b1f3224c667801624ea198242013f3164d33bd8a7362a3e4cd0ff77720 Oct 30 18:01:09.048687 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:09.048656 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:01:09.048767 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:09.048719 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:01:09.048767 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:09.048752 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:01:09.063265 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:09.063238 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn"] Oct 30 18:01:09.065063 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:01:09.065033 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77c24c9_98c1_43c8_9460_e5b3c509c6d3.slice/crio-3b510f2f29581cc23659b9c733d7f479b9c2c16738decac5b9bf42444ab28dcb WatchSource:0}: Error finding container 3b510f2f29581cc23659b9c733d7f479b9c2c16738decac5b9bf42444ab28dcb: Status 404 returned error can't find the container with id 3b510f2f29581cc23659b9c733d7f479b9c2c16738decac5b9bf42444ab28dcb Oct 30 18:01:09.067870 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:09.067836 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:01:09.067965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:09.067914 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:01:09.068027 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:09.067964 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:01:09.075998 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:09.075975 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:09.092543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:09.092526 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:01:09.110324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:09.110306 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:01:09.157566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:09.157486 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:01:09.563788 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:09.563732 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" event={"ID":"c77c24c9-98c1-43c8-9460-e5b3c509c6d3","Type":"ContainerStarted","Data":"85df9046514b1f04caaa202dbd4c850187e0f632f663a5e2eebcdedff28170d5"} Oct 30 18:01:09.564259 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:09.563801 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" event={"ID":"c77c24c9-98c1-43c8-9460-e5b3c509c6d3","Type":"ContainerStarted","Data":"3b510f2f29581cc23659b9c733d7f479b9c2c16738decac5b9bf42444ab28dcb"} Oct 30 18:01:09.565525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:09.565467 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" event={"ID":"5bf7d070-07da-4ad5-8464-e39192f8e23e","Type":"ContainerStarted","Data":"2ec99e2ec8f1eda19858be1f8b34e7895b4a91b5daf5d7bf9a3208b65516bb93"} Oct 30 18:01:09.565635 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:09.565536 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" event={"ID":"5bf7d070-07da-4ad5-8464-e39192f8e23e","Type":"ContainerStarted","Data":"b67cf4b1f3224c667801624ea198242013f3164d33bd8a7362a3e4cd0ff77720"} Oct 30 18:01:09.582740 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:09.582717 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" Oct 30 18:01:09.609123 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:09.609062 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" podStartSLOduration=1.609042659 podStartE2EDuration="1.609042659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:09.607525407 +0000 UTC m=+530.323095290" watchObservedRunningTime="2026-10-30 18:01:09.609042659 +0000 UTC m=+530.324612535" Oct 30 18:01:09.706527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:09.702949 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:09.745542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:09.740161 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:10.697924 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:10.697887 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-x4522 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 30 18:01:10.698568 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:10.698537 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" podUID="5bf7d070-07da-4ad5-8464-e39192f8e23e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 30 18:01:10.741485 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:10.741455 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-wjstn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 18:01:10.741674 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:10.741523 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" podUID="c77c24c9-98c1-43c8-9460-e5b3c509c6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 18:01:11.692920 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:11.692881 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-x4522 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 30 18:01:11.693171 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:11.693141 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" podUID="5bf7d070-07da-4ad5-8464-e39192f8e23e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 30 18:01:11.740017 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:11.739989 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-wjstn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 18:01:11.740428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:11.740406 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" podUID="c77c24c9-98c1-43c8-9460-e5b3c509c6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 18:01:12.317529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:12.317454 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:12.692977 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:12.692888 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-x4522 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 30 18:01:12.693231 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:12.693203 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" podUID="5bf7d070-07da-4ad5-8464-e39192f8e23e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 30 18:01:12.740176 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:12.740142 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-wjstn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 18:01:12.740559 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:12.740188 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" podUID="c77c24c9-98c1-43c8-9460-e5b3c509c6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 18:01:12.918529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:12.917575 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:12.941909 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:12.941882 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:12.983074 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:12.982971 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" podStartSLOduration=4.982957544 podStartE2EDuration="4.982957544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:09.702891221 +0000 UTC m=+530.418461110" watchObservedRunningTime="2026-10-30 18:01:12.982957544 +0000 UTC m=+533.698527418" Oct 30 18:01:13.614527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:13.609781 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" Oct 30 18:01:13.692864 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:13.692832 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-x4522 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 30 18:01:13.693120 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:13.693095 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" podUID="5bf7d070-07da-4ad5-8464-e39192f8e23e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 30 18:01:13.741529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:13.740964 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-wjstn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 18:01:13.741529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:13.741023 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" podUID="c77c24c9-98c1-43c8-9460-e5b3c509c6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 18:01:14.692904 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:14.692864 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-x4522 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 30 18:01:14.693202 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:14.693169 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" podUID="5bf7d070-07da-4ad5-8464-e39192f8e23e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 30 18:01:14.740926 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:14.740886 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-wjstn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 18:01:14.741066 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:14.740955 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" podUID="c77c24c9-98c1-43c8-9460-e5b3c509c6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 18:01:15.690781 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:15.690742 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-x4522 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 30 18:01:15.691282 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:15.690818 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" podUID="5bf7d070-07da-4ad5-8464-e39192f8e23e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 30 18:01:15.740535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:15.740495 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-wjstn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 18:01:15.740701 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:15.740551 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" podUID="c77c24c9-98c1-43c8-9460-e5b3c509c6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 18:01:16.690695 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:16.690662 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-x4522 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 30 18:01:16.690877 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:16.690714 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" podUID="5bf7d070-07da-4ad5-8464-e39192f8e23e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 30 18:01:16.740135 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:16.740100 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-wjstn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 18:01:16.740295 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:16.740149 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" podUID="c77c24c9-98c1-43c8-9460-e5b3c509c6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 18:01:17.076130 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:17.076096 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:17.096702 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:17.096671 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:17.612368 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:17.612342 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:01:17.694659 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:17.694632 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:17.695115 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:17.694922 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:17.695743 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:17.695724 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:01:17.743650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:17.743619 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:17.744007 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:17.743984 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:17.744651 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:17.744631 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" Oct 30 18:01:21.929159 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:01:21.929121 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:01:32.723399 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:32.723357 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:32.876131 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:32.876101 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:34.176577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:34.176535 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:34.338954 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:01:34.338911 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:51.928185 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:01:51.928151 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:02:12.453545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:02:12.453453 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:19.464121 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:02:19.464096 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 18:02:19.464603 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:02:19.464104 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 18:02:21.928895 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:02:21.928861 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:02:51.928467 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:02:51.928424 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:03:00.258830 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:00.258795 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:00.409883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:00.409850 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:00.557027 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:00.556996 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:00.787047 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:00.787010 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:00.873349 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:00.873282 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p"] Oct 30 18:03:00.875435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:00.875421 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:00.897153 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:00.897128 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ce05c7ef-openshift-default-dockercfg-r5trp\"" Oct 30 18:03:00.912193 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:00.912174 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p"] Oct 30 18:03:01.010145 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.010108 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:01.045414 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.045384 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt"] Oct 30 18:03:01.047980 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.047966 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.052048 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.052025 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-workload-socket\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.052183 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.052090 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76e15ae2-c331-4780-b383-d99fbd721312-istio-podinfo\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.052183 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.052123 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-workload-certs\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.052183 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.052140 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cznh\" (UniqueName: \"kubernetes.io/projected/76e15ae2-c331-4780-b383-d99fbd721312-kube-api-access-8cznh\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.052183 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.052158 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-istio-data\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.052183 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.052174 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76e15ae2-c331-4780-b383-d99fbd721312-istiod-ca-cert\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.052365 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.052190 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-istio-envoy\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.052365 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.052233 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-credential-socket\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.052365 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.052256 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76e15ae2-c331-4780-b383-d99fbd721312-istio-token\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.066947 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.066930 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a8c61e05-openshift-default-dockercfg-wbshc\"" Oct 30 18:03:01.090690 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.090666 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt"] Oct 30 18:03:01.152945 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.152888 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76e15ae2-c331-4780-b383-d99fbd721312-istiod-ca-cert\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.152945 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.152919 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76e15ae2-c331-4780-b383-d99fbd721312-istio-podinfo\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.152945 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.152940 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cznh\" (UniqueName: \"kubernetes.io/projected/76e15ae2-c331-4780-b383-d99fbd721312-kube-api-access-8cznh\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.153178 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.153057 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istiod-ca-cert\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.153178 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.153102 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-workload-socket\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.153277 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.153233 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wndkp\" (UniqueName: \"kubernetes.io/projected/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-kube-api-access-wndkp\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.153339 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.153274 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-istio-envoy\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.153339 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.153297 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-workload-certs\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.153430 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.153385 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-credential-socket\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.153465 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.153450 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-istio-data\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.153538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.153479 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-workload-certs\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.153589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.153552 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76e15ae2-c331-4780-b383-d99fbd721312-istio-token\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.153642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.153584 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-token\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.153642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.153614 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-credential-socket\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.153719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.153644 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76e15ae2-c331-4780-b383-d99fbd721312-istiod-ca-cert\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.153789 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.153773 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-workload-certs\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.153862 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.153835 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-envoy\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.153983 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.153881 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-data\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.153983 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.153946 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-workload-socket\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.154066 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.153986 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-podinfo\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.154136 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.154108 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-credential-socket\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.154282 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.154267 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-workload-socket\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.154320 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.154279 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-istio-data\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.155610 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.155589 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-istio-envoy\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.155762 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.155734 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76e15ae2-c331-4780-b383-d99fbd721312-istio-podinfo\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.171910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.171885 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:01.176227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.176210 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76e15ae2-c331-4780-b383-d99fbd721312-istio-token\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.177025 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.177008 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cznh\" (UniqueName: \"kubernetes.io/projected/76e15ae2-c331-4780-b383-d99fbd721312-kube-api-access-8cznh\") pod \"e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.186705 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.186688 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:01.255215 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.255189 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istiod-ca-cert\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.255328 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.255225 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-workload-socket\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.255328 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.255270 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wndkp\" (UniqueName: \"kubernetes.io/projected/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-kube-api-access-wndkp\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.255328 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.255295 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-workload-certs\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.255328 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.255312 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-credential-socket\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.255464 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.255345 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-token\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.255464 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.255362 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-envoy\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.255464 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.255380 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-data\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.255464 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.255407 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-podinfo\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.256133 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.256102 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-credential-socket\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.256201 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.256134 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istiod-ca-cert\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.256431 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.256411 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-workload-socket\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.256696 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.256677 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-workload-certs\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.257111 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.257088 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-data\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.258707 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.258686 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-podinfo\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.258965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.258941 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-envoy\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.285632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.285581 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wndkp\" (UniqueName: \"kubernetes.io/projected/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-kube-api-access-wndkp\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.285748 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.285690 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-token\") pod \"e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.287369 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.287344 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf"] Oct 30 18:03:01.291907 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.291887 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.307670 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.307649 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-fd500666-gw-openshift-default-dockercfg-fn8ld\"" Oct 30 18:03:01.312842 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.312821 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf"] Oct 30 18:03:01.340633 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.340602 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:01.360619 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.360599 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:01.468525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.463746 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-credential-socket\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.468525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.463841 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2gnz\" (UniqueName: \"kubernetes.io/projected/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-kube-api-access-h2gnz\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.468525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.463898 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-workload-certs\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.468525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.463942 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-workload-socket\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.468525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.464010 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istiod-ca-cert\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.468525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.464041 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-podinfo\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.468525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.464060 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-data\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.468525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.464079 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-envoy\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.468525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.464096 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-token\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.518492 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.517568 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:01.565092 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.565058 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-workload-socket\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.565266 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.565157 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istiod-ca-cert\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.565266 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.565209 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-podinfo\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.565266 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.565238 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-data\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.565453 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.565269 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-envoy\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.565453 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.565297 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-token\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.565608 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.565497 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-credential-socket\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.565608 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.565575 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-workload-socket\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.565724 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.565646 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2gnz\" (UniqueName: \"kubernetes.io/projected/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-kube-api-access-h2gnz\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.565724 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.565667 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-data\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.565826 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.565729 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-workload-certs\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.566371 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.566104 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istiod-ca-cert\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.566523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.566369 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-credential-socket\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.566523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.566447 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-workload-certs\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.568446 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.568425 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-envoy\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.568636 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.568615 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p"] Oct 30 18:03:01.568811 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.568794 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-podinfo\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.570043 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:03:01.570023 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e15ae2_c331_4780_b383_d99fbd721312.slice/crio-5de2efe09e466b8a01a2fc2121dce5cbde1b6eda4e93f445043ad95790c7b20b WatchSource:0}: Error finding container 5de2efe09e466b8a01a2fc2121dce5cbde1b6eda4e93f445043ad95790c7b20b: Status 404 returned error can't find the container with id 5de2efe09e466b8a01a2fc2121dce5cbde1b6eda4e93f445043ad95790c7b20b Oct 30 18:03:01.572233 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.572207 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:01.572309 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.572264 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:01.572343 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.572308 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:01.643548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.643491 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2gnz\" (UniqueName: \"kubernetes.io/projected/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-kube-api-access-h2gnz\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.651993 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.651963 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-token\") pod \"e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:01.676806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.676670 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt"] Oct 30 18:03:01.679383 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:03:01.679354 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d00243_a6a6_438f_ab25_9ae2ac5e75bd.slice/crio-6db5471efd2999fe92af49354b4187ebf87ceccad2a3ba6719ea9fde13818f96 WatchSource:0}: Error finding container 6db5471efd2999fe92af49354b4187ebf87ceccad2a3ba6719ea9fde13818f96: Status 404 returned error can't find the container with id 6db5471efd2999fe92af49354b4187ebf87ceccad2a3ba6719ea9fde13818f96 Oct 30 18:03:01.681751 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.681713 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:01.681855 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.681793 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:01.681916 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.681834 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:01.905166 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:01.905129 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:02.061453 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:02.061417 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" event={"ID":"76e15ae2-c331-4780-b383-d99fbd721312","Type":"ContainerStarted","Data":"36ccc03ba70c43b3ffc042b6e8ff799d8a322723f9c2475694989a2310dfe8f8"} Oct 30 18:03:02.061658 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:02.061481 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" event={"ID":"76e15ae2-c331-4780-b383-d99fbd721312","Type":"ContainerStarted","Data":"5de2efe09e466b8a01a2fc2121dce5cbde1b6eda4e93f445043ad95790c7b20b"} Oct 30 18:03:02.062969 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:02.062940 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" event={"ID":"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd","Type":"ContainerStarted","Data":"452b45a9d92b4b5af4aa38811bb5bd9d8973cd2a79b3e94b4dcd169ac25c0bfa"} Oct 30 18:03:02.063097 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:02.062979 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" event={"ID":"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd","Type":"ContainerStarted","Data":"6db5471efd2999fe92af49354b4187ebf87ceccad2a3ba6719ea9fde13818f96"} Oct 30 18:03:02.108178 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:02.108115 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" podStartSLOduration=2.1080967250000002 podStartE2EDuration="2.108096725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:02.106849827 +0000 UTC m=+642.822419702" watchObservedRunningTime="2026-10-30 18:03:02.108096725 +0000 UTC m=+642.823666598" Oct 30 18:03:02.154940 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:02.154902 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" podStartSLOduration=1.154890726 podStartE2EDuration="1.154890726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:02.154470865 +0000 UTC m=+642.870040739" watchObservedRunningTime="2026-10-30 18:03:02.154890726 +0000 UTC m=+642.870460599" Oct 30 18:03:02.187907 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:02.187885 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:02.193064 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:02.193046 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:02.222791 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:02.222739 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf"] Oct 30 18:03:02.225960 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:03:02.225938 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae1fe54_cb96_4cf9_9d30_5d5d67f3b25c.slice/crio-98e05d212eb7f120f5019088d71f35cc5ea7d0ee87e2c9fe5d43d88184072aca WatchSource:0}: Error finding container 98e05d212eb7f120f5019088d71f35cc5ea7d0ee87e2c9fe5d43d88184072aca: Status 404 returned error can't find the container with id 98e05d212eb7f120f5019088d71f35cc5ea7d0ee87e2c9fe5d43d88184072aca Oct 30 18:03:02.228686 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:02.228657 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:02.228751 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:02.228731 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:02.228789 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:02.228776 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:02.361553 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:02.361524 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:02.367744 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:02.367722 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:03.069715 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:03.069674 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" event={"ID":"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c","Type":"ContainerStarted","Data":"07f0d6b90e91fa43badf59c24d209b41e6553e3888dd31e8ba45951cc2df638b"} Oct 30 18:03:03.069907 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:03.069742 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" event={"ID":"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c","Type":"ContainerStarted","Data":"98e05d212eb7f120f5019088d71f35cc5ea7d0ee87e2c9fe5d43d88184072aca"} Oct 30 18:03:03.070164 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:03.070137 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:03.070285 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:03.070192 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:03.070697 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:03.070678 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:03.070793 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:03.070743 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:03.115490 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:03.115447 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" podStartSLOduration=2.115432591 podStartE2EDuration="2.115432591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:03.113416663 +0000 UTC m=+643.828986537" watchObservedRunningTime="2026-10-30 18:03:03.115432591 +0000 UTC m=+643.831002466" Oct 30 18:03:03.231739 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:03.231701 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:03.793303 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:03.793267 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:03.906013 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:03.905985 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:03.911031 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:03.911009 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:03.975997 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:03.975966 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:04.074076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:04.074008 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:04.074805 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:04.074788 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:04.133663 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:04.133637 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:05.546282 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:05.546255 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:05.716493 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:05.716463 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:06.424736 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:06.424708 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:06.566145 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:06.566103 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:07.347017 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:07.346983 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:07.495796 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:07.495758 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:07.669166 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:07.669073 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:07.993423 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:07.993341 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:08.144088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.144049 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:08.321772 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.321736 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:08.472655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.472613 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:08.735739 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.735668 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb"] Oct 30 18:03:08.738722 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.738700 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.758318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.758297 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-5e0da928-openshift-default-dockercfg-m5mtf\"" Oct 30 18:03:08.769958 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.769932 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb"] Oct 30 18:03:08.812653 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.812627 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:08.851794 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.851771 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdwcv\" (UniqueName: \"kubernetes.io/projected/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-kube-api-access-zdwcv\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.851895 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.851807 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-workload-socket\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.851895 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.851845 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istiod-ca-cert\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.851970 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.851895 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-credential-socket\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.852002 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.851970 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-data\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.852036 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.852003 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-workload-certs\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.852036 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.852021 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-podinfo\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.852135 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.852090 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-envoy\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.852135 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.852130 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-token\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.952982 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.952948 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-data\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.953168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.953018 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-workload-certs\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.953168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.953048 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-podinfo\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.953168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.953094 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-envoy\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.953168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.953138 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-token\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.953385 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.953175 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdwcv\" (UniqueName: \"kubernetes.io/projected/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-kube-api-access-zdwcv\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.953514 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.953476 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-workload-certs\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.953596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.953476 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-data\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.953681 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.953662 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-workload-socket\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.953735 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.953719 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istiod-ca-cert\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.953788 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.953778 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-credential-socket\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.954047 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.954028 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-credential-socket\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.954168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.954146 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-workload-socket\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.954710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.954688 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istiod-ca-cert\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.955810 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.955787 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-envoy\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.956133 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.956114 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-podinfo\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.977871 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.977850 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-token\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:08.980778 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:08.980759 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdwcv\" (UniqueName: \"kubernetes.io/projected/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-kube-api-access-zdwcv\") pod \"e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:09.017813 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:09.017763 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:09.051496 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:09.051458 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:09.382762 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:03:09.382720 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c7ffd4_3ef5_4774_a9d9_a29819f2f855.slice/crio-fc553b1eecefda51692c6fd732154cb37309d65bc76a64a739c83911bcdceb51 WatchSource:0}: Error finding container fc553b1eecefda51692c6fd732154cb37309d65bc76a64a739c83911bcdceb51: Status 404 returned error can't find the container with id fc553b1eecefda51692c6fd732154cb37309d65bc76a64a739c83911bcdceb51 Oct 30 18:03:09.384796 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:09.384633 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:09.387000 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:09.386962 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb"] Oct 30 18:03:09.387264 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:09.387230 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:09.387367 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:09.387308 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:09.387367 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:09.387353 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:10.048734 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:10.048700 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:03:10.110815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:10.110776 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" event={"ID":"87c7ffd4-3ef5-4774-a9d9-a29819f2f855","Type":"ContainerStarted","Data":"26ffb791a2d79b2211236c2471499bbb8aedb696d78ddd36bd710e0f0b0d2061"} Oct 30 18:03:10.110997 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:10.110833 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" event={"ID":"87c7ffd4-3ef5-4774-a9d9-a29819f2f855","Type":"ContainerStarted","Data":"fc553b1eecefda51692c6fd732154cb37309d65bc76a64a739c83911bcdceb51"} Oct 30 18:03:11.052223 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:11.052188 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:11.057094 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:11.057070 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:11.116262 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:11.116241 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:11.116962 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:11.116947 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:11.169260 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:11.169216 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" podStartSLOduration=3.169194105 podStartE2EDuration="3.169194105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:10.178258515 +0000 UTC m=+650.893828389" watchObservedRunningTime="2026-10-30 18:03:11.169194105 +0000 UTC m=+651.884763979" Oct 30 18:03:11.288629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:11.288588 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:11.446357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:11.446165 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:11.829901 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:11.829864 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:11.981724 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:11.981693 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:12.608413 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:12.608379 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:13.240250 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:13.240217 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:13.428804 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:13.426996 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:14.315249 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:14.315216 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:14.486909 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:14.486877 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:15.176648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:15.176616 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:16.462886 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:16.462855 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:03:17.045339 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:17.045025 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:18.941988 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:18.941955 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:19.088668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:19.088599 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:19.120002 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:19.119628 4603 patch_prober.go:29] interesting pod/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:03:19.120002 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:19.119701 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" podUID="e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:03:20.755953 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:20.755915 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:21.441752 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:21.440343 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:03:21.962435 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:03:21.962399 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:03:22.173194 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.173120 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:22.485091 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.485058 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:22.581530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.577942 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk"] Oct 30 18:03:22.687359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.584722 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.687359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.602533 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a9a40825-openshift-default-dockercfg-rhrqv\"" Oct 30 18:03:22.687359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.603694 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lqkb\" (UniqueName: \"kubernetes.io/projected/e355edcc-648d-492b-9665-2107f84afb63-kube-api-access-7lqkb\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.687359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.603732 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-workload-socket\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.687359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.603784 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e355edcc-648d-492b-9665-2107f84afb63-istiod-ca-cert\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.687359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.603822 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-credential-socket\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.687359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.603862 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-istio-envoy\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.687359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.603889 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-istio-data\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.687359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.603924 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e355edcc-648d-492b-9665-2107f84afb63-istio-token\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.687359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.603997 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e355edcc-648d-492b-9665-2107f84afb63-istio-podinfo\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.687359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.604036 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-workload-certs\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.687359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.620471 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk"] Oct 30 18:03:22.714919 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.714871 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-istio-envoy\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.714919 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.714923 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-istio-data\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.715207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.714960 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e355edcc-648d-492b-9665-2107f84afb63-istio-token\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.715207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.715036 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e355edcc-648d-492b-9665-2107f84afb63-istio-podinfo\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.715207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.715081 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-workload-certs\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.715207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.715134 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lqkb\" (UniqueName: \"kubernetes.io/projected/e355edcc-648d-492b-9665-2107f84afb63-kube-api-access-7lqkb\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.715207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.715189 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-workload-socket\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.715484 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.715239 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e355edcc-648d-492b-9665-2107f84afb63-istiod-ca-cert\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.715484 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.715279 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-credential-socket\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.748578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.715739 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-credential-socket\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.749163 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.749133 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-workload-certs\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.753526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.749643 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-istio-data\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.779544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.755352 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-workload-socket\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.779544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.767307 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e355edcc-648d-492b-9665-2107f84afb63-istio-podinfo\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.793529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.793043 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e355edcc-648d-492b-9665-2107f84afb63-istio-token\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.804180 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.804153 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:22.838800 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.824295 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lqkb\" (UniqueName: \"kubernetes.io/projected/e355edcc-648d-492b-9665-2107f84afb63-kube-api-access-7lqkb\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.838800 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.826803 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-istio-envoy\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.861564 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.843270 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e355edcc-648d-492b-9665-2107f84afb63-istiod-ca-cert\") pod \"e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:22.981466 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.981383 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:22.996089 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:22.995972 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:23.471900 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:23.471789 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk"] Oct 30 18:03:23.474384 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:03:23.474360 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode355edcc_648d_492b_9665_2107f84afb63.slice/crio-9a5f6f1967b58bfe1df2a82e45e0bad140ecceef73f0bb53d66b45bcb66dc93a WatchSource:0}: Error finding container 9a5f6f1967b58bfe1df2a82e45e0bad140ecceef73f0bb53d66b45bcb66dc93a: Status 404 returned error can't find the container with id 9a5f6f1967b58bfe1df2a82e45e0bad140ecceef73f0bb53d66b45bcb66dc93a Oct 30 18:03:23.476535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:23.476481 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:23.476627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:23.476579 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:23.476687 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:23.476626 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:24.238581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:24.228567 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" event={"ID":"e355edcc-648d-492b-9665-2107f84afb63","Type":"ContainerStarted","Data":"2258b6a8808bef06a850862f288d2b4769736952d4d1efe9f87a65145af06b69"} Oct 30 18:03:24.238581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:24.228636 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" event={"ID":"e355edcc-648d-492b-9665-2107f84afb63","Type":"ContainerStarted","Data":"9a5f6f1967b58bfe1df2a82e45e0bad140ecceef73f0bb53d66b45bcb66dc93a"} Oct 30 18:03:24.240709 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:24.239770 4603 generic.go:356] "Generic (PLEG): container finished" podID="e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" containerID="452b45a9d92b4b5af4aa38811bb5bd9d8973cd2a79b3e94b4dcd169ac25c0bfa" exitCode=0 Oct 30 18:03:24.240709 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:24.239839 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" event={"ID":"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd","Type":"ContainerDied","Data":"452b45a9d92b4b5af4aa38811bb5bd9d8973cd2a79b3e94b4dcd169ac25c0bfa"} Oct 30 18:03:24.240709 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:24.240252 4603 scope.go:122] "RemoveContainer" containerID="452b45a9d92b4b5af4aa38811bb5bd9d8973cd2a79b3e94b4dcd169ac25c0bfa" Oct 30 18:03:24.243269 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:24.242485 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:24.243269 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:24.242590 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:24.243269 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:24.242638 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:24.287261 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:24.287222 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" podStartSLOduration=2.287208684 podStartE2EDuration="2.287208684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:24.285893913 +0000 UTC m=+665.001463788" watchObservedRunningTime="2026-10-30 18:03:24.287208684 +0000 UTC m=+665.002778555" Oct 30 18:03:24.456425 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:24.456308 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:25.007524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:24.996857 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:25.026979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:25.025797 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:25.076363 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:25.076327 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l"] Oct 30 18:03:25.079570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:25.079545 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l" Oct 30 18:03:25.110524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:25.106881 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:03:25.110524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:25.106970 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:03:25.110524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:25.110193 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l"] Oct 30 18:03:25.183637 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:25.183598 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2trd\" (UniqueName: \"kubernetes.io/projected/f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a-kube-api-access-c2trd\") pod \"maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l\" (UID: \"f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l" Oct 30 18:03:25.183836 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:25.183645 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l\" (UID: \"f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l" Oct 30 18:03:25.183836 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:25.183703 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l\" (UID: \"f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l" Oct 30 18:03:25.250297 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:25.250260 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" event={"ID":"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd","Type":"ContainerStarted","Data":"4ff976d9c9f218b3737c21f0833730cad18723ea25f9ee0a855cee79e90b3317"} Oct 30 18:03:25.250818 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:25.250802 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:25.252218 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:25.252203 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:25.284922 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:25.284900 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l\" (UID: \"f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l" Oct 30 18:03:25.285069 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:25.285028 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2trd\" (UniqueName: \"kubernetes.io/projected/f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a-kube-api-access-c2trd\") pod \"maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l\" (UID: \"f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l" Oct 30 18:03:25.285069 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:25.285062 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l\" (UID: \"f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l" Oct 30 18:03:25.287087 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:25.287026 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l\" (UID: \"f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l" Oct 30 18:03:25.316527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:25.315943 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l\" (UID: \"f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l" Oct 30 18:03:25.335538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:25.325372 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2trd\" (UniqueName: \"kubernetes.io/projected/f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a-kube-api-access-c2trd\") pod \"maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l\" (UID: \"f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l" Oct 30 18:03:25.361040 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:25.361001 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:25.361223 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:25.361115 4603 patch_prober.go:29] interesting pod/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 18:03:25.361223 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:25.361152 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" podUID="e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 18:03:25.433566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:25.423657 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l" Oct 30 18:03:25.818601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:25.814776 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l"] Oct 30 18:03:26.036819 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:26.036783 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:26.185141 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:26.185059 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:26.257390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:26.257350 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l" event={"ID":"f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a","Type":"ContainerStarted","Data":"36691d117cb467a1166de1a3148fc67214a39a092179c7f7804e8ed91bd7e1b4"} Oct 30 18:03:26.354542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:26.343172 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:26.373219 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:26.373004 4603 patch_prober.go:29] interesting pod/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 18:03:26.373219 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:26.373058 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" podUID="e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 18:03:27.362006 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:27.361964 4603 patch_prober.go:29] interesting pod/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 18:03:27.362410 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:27.362030 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" podUID="e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 18:03:27.832174 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:27.832129 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:27.985743 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:27.985708 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:28.166572 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:28.166462 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:28.361901 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:28.361862 4603 patch_prober.go:29] interesting pod/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 18:03:28.362063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:28.361933 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" podUID="e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 18:03:29.362378 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:29.361890 4603 patch_prober.go:29] interesting pod/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 18:03:29.362378 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:29.361953 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" podUID="e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 18:03:29.919544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:29.918994 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:30.361664 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:30.361623 4603 patch_prober.go:29] interesting pod/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 18:03:30.361841 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:30.361698 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" podUID="e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 18:03:31.809806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:31.288014 4603 generic.go:356] "Generic (PLEG): container finished" podID="76e15ae2-c331-4780-b383-d99fbd721312" containerID="36ccc03ba70c43b3ffc042b6e8ff799d8a322723f9c2475694989a2310dfe8f8" exitCode=0 Oct 30 18:03:31.809806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:31.288095 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" event={"ID":"76e15ae2-c331-4780-b383-d99fbd721312","Type":"ContainerDied","Data":"36ccc03ba70c43b3ffc042b6e8ff799d8a322723f9c2475694989a2310dfe8f8"} Oct 30 18:03:31.809806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:31.288536 4603 scope.go:122] "RemoveContainer" containerID="36ccc03ba70c43b3ffc042b6e8ff799d8a322723f9c2475694989a2310dfe8f8" Oct 30 18:03:31.809806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:31.290116 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:31.809806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:31.290191 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l" event={"ID":"f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a","Type":"ContainerStarted","Data":"63f915de7d4e62444545babb63dfa0ff476258dad47682955ffeb8eba7e2f435"} Oct 30 18:03:31.809806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:31.290209 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:31.809806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:31.290270 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:31.809806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:31.365930 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:31.809806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:31.366212 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:31.809806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:31.366961 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:31.809806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:31.384115 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l" podStartSLOduration=1.593502779 podStartE2EDuration="6.384098885s" totalImagesPullingTime="4.790596106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:31.383279164 +0000 UTC m=+672.098849039" watchObservedRunningTime="2026-10-30 18:03:31.384098885 +0000 UTC m=+672.099668761" Oct 30 18:03:31.846110 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:31.825284 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:32.319610 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:32.319579 4603 generic.go:356] "Generic (PLEG): container finished" podID="3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" containerID="07f0d6b90e91fa43badf59c24d209b41e6553e3888dd31e8ba45951cc2df638b" exitCode=0 Oct 30 18:03:32.319808 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:32.319655 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" event={"ID":"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c","Type":"ContainerDied","Data":"07f0d6b90e91fa43badf59c24d209b41e6553e3888dd31e8ba45951cc2df638b"} Oct 30 18:03:32.320089 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:32.320068 4603 scope.go:122] "RemoveContainer" containerID="07f0d6b90e91fa43badf59c24d209b41e6553e3888dd31e8ba45951cc2df638b" Oct 30 18:03:32.321381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:32.321348 4603 generic.go:356] "Generic (PLEG): container finished" podID="f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a" containerID="63f915de7d4e62444545babb63dfa0ff476258dad47682955ffeb8eba7e2f435" exitCode=0 Oct 30 18:03:32.321495 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:32.321416 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:32.321495 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:32.321479 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:32.321495 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:32.321426 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l" event={"ID":"f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a","Type":"ContainerDied","Data":"63f915de7d4e62444545babb63dfa0ff476258dad47682955ffeb8eba7e2f435"} Oct 30 18:03:32.321652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:32.321537 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:32.323447 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:32.323426 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" event={"ID":"76e15ae2-c331-4780-b383-d99fbd721312","Type":"ContainerStarted","Data":"9b7b91be8399307fbbd8c6f1832b19f1617ff014dcb3865218e9c7f951ea7589"} Oct 30 18:03:33.070200 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.070152 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:33.187580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.187544 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:33.196540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.192630 4603 patch_prober.go:29] interesting pod/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 18:03:33.196540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.192699 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" podUID="76e15ae2-c331-4780-b383-d99fbd721312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 18:03:33.344397 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.344319 4603 generic.go:356] "Generic (PLEG): container finished" podID="87c7ffd4-3ef5-4774-a9d9-a29819f2f855" containerID="26ffb791a2d79b2211236c2471499bbb8aedb696d78ddd36bd710e0f0b0d2061" exitCode=0 Oct 30 18:03:33.344558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.344392 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" event={"ID":"87c7ffd4-3ef5-4774-a9d9-a29819f2f855","Type":"ContainerDied","Data":"26ffb791a2d79b2211236c2471499bbb8aedb696d78ddd36bd710e0f0b0d2061"} Oct 30 18:03:33.344915 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.344898 4603 scope.go:122] "RemoveContainer" containerID="26ffb791a2d79b2211236c2471499bbb8aedb696d78ddd36bd710e0f0b0d2061" Oct 30 18:03:33.346885 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.346860 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:33.346994 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.346912 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" event={"ID":"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c","Type":"ContainerStarted","Data":"7c1ffa567f8498caa5a477fa189b5f36d057dc6cead05d14d881a89dbb80a07b"} Oct 30 18:03:33.347039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.347000 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:33.347090 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.347037 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:33.686540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.680231 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l" Oct 30 18:03:33.763539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.763474 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:33.790898 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.790861 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a-maas-api-xks-ca\") pod \"f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a\" (UID: \"f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a\") " Oct 30 18:03:33.798575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.798546 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a-maas-api-openshift-ca\") pod \"f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a\" (UID: \"f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a\") " Oct 30 18:03:33.813528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.808708 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a-kube-api-access-c2trd\" (UniqueName: \"kubernetes.io/projected/f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a-kube-api-access-c2trd\") pod \"f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a\" (UID: \"f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a\") " Oct 30 18:03:33.838737 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.838686 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a-maas-api-openshift-ca" pod "f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a" (UID: "f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:33.865307 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.865251 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a-maas-api-xks-ca" pod "f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a" (UID: "f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:03:33.868091 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.868054 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a-kube-api-access-c2trd" pod "f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a" (UID: "f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a"). InnerVolumeSpecName "kube-api-access-c2trd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:33.912581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.911910 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:33.915853 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.915491 4603 patch_prober.go:29] interesting pod/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 18:03:33.915853 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.915571 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" podUID="3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 18:03:33.931538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.931160 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2trd\" (UniqueName: \"kubernetes.io/projected/f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a-kube-api-access-c2trd\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:33.931538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.931199 4603 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a-maas-api-xks-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:33.931538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:33.931215 4603 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a-maas-api-openshift-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:34.074223 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:34.074197 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:34.187911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:34.187832 4603 patch_prober.go:29] interesting pod/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 18:03:34.187911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:34.187885 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" podUID="76e15ae2-c331-4780-b383-d99fbd721312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 18:03:34.366132 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:34.366093 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" event={"ID":"87c7ffd4-3ef5-4774-a9d9-a29819f2f855","Type":"ContainerStarted","Data":"4d1637b1c34ad94d4edc9b953726d8b8710f389c51e7cc642412c7cb0d2e8769"} Oct 30 18:03:34.369228 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:34.369205 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l" Oct 30 18:03:34.369472 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:34.369446 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l" event={"ID":"f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a","Type":"ContainerDied","Data":"36691d117cb467a1166de1a3148fc67214a39a092179c7f7804e8ed91bd7e1b4"} Oct 30 18:03:34.369579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:34.369520 4603 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36691d117cb467a1166de1a3148fc67214a39a092179c7f7804e8ed91bd7e1b4" Oct 30 18:03:34.910530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:34.906978 4603 patch_prober.go:29] interesting pod/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 18:03:34.910530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:34.907041 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" podUID="3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 18:03:35.052102 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.051673 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:35.052525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.052478 4603 patch_prober.go:29] interesting pod/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 18:03:35.052705 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.052682 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" podUID="87c7ffd4-3ef5-4774-a9d9-a29819f2f855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 18:03:35.187939 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.187863 4603 patch_prober.go:29] interesting pod/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 18:03:35.188383 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.188357 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" podUID="76e15ae2-c331-4780-b383-d99fbd721312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 18:03:35.468609 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.468479 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:35.527529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.508882 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz"] Oct 30 18:03:35.527529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.509480 4603 cpu_manager.go:414] "containerMap: removing container" podUID="f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a" containerName="revoke-keys" Oct 30 18:03:35.533535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.527848 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a" containerName="revoke-keys" Oct 30 18:03:35.533535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.528070 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a" containerName="revoke-keys" Oct 30 18:03:35.602637 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.602600 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" Oct 30 18:03:35.652538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.630976 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz"] Oct 30 18:03:35.671534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.669530 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:35.740582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.737312 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a9a40825-serving-cert\"" Oct 30 18:03:35.804200 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.803240 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-metrics-certs\") pod \"maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz\" (UID: \"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" Oct 30 18:03:35.804200 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.803365 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9v2q\" (UniqueName: \"kubernetes.io/projected/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-kube-api-access-z9v2q\") pod \"maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz\" (UID: \"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" Oct 30 18:03:35.804200 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.803410 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-maas-api-tls\") pod \"maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz\" (UID: \"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" Oct 30 18:03:35.807895 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.806693 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-openshift-service-ca\") pod \"maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz\" (UID: \"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" Oct 30 18:03:35.895070 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.895032 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:35.908081 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.908041 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9v2q\" (UniqueName: \"kubernetes.io/projected/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-kube-api-access-z9v2q\") pod \"maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz\" (UID: \"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" Oct 30 18:03:35.915528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.911604 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-maas-api-tls\") pod \"maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz\" (UID: \"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" Oct 30 18:03:35.940482 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.940448 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-openshift-service-ca\") pod \"maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz\" (UID: \"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" Oct 30 18:03:35.967710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.967679 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-metrics-certs\") pod \"maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz\" (UID: \"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" Oct 30 18:03:35.968372 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.945272 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-openshift-service-ca\") pod \"maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz\" (UID: \"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" Oct 30 18:03:35.968535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.946604 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-maas-api-tls\") pod \"maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz\" (UID: \"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" Oct 30 18:03:35.968760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.947166 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9v2q\" (UniqueName: \"kubernetes.io/projected/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-kube-api-access-z9v2q\") pod \"maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz\" (UID: \"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" Oct 30 18:03:35.968885 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.908290 4603 patch_prober.go:29] interesting pod/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 18:03:35.969033 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.969008 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" podUID="3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 18:03:35.972830 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:35.972805 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-metrics-certs\") pod \"maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz\" (UID: \"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" Oct 30 18:03:36.052294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:36.052057 4603 patch_prober.go:29] interesting pod/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 18:03:36.052294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:36.052146 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" podUID="87c7ffd4-3ef5-4774-a9d9-a29819f2f855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 18:03:36.192542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:36.191721 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" Oct 30 18:03:36.193044 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:36.192601 4603 patch_prober.go:29] interesting pod/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 18:03:36.193044 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:36.192651 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" podUID="76e15ae2-c331-4780-b383-d99fbd721312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 18:03:36.928531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:36.921533 4603 patch_prober.go:29] interesting pod/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 18:03:36.928531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:36.921596 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" podUID="3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 18:03:36.949814 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:36.948988 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz"] Oct 30 18:03:37.062531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:37.059885 4603 patch_prober.go:29] interesting pod/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 18:03:37.062531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:37.059950 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" podUID="87c7ffd4-3ef5-4774-a9d9-a29819f2f855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 18:03:37.191584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:37.190536 4603 patch_prober.go:29] interesting pod/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 18:03:37.191584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:37.190615 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" podUID="76e15ae2-c331-4780-b383-d99fbd721312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 18:03:37.429365 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:37.429331 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" event={"ID":"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61","Type":"ContainerStarted","Data":"ccee212f0157977e8ac97e8cbad9ff12718b5c6ce29be25d60f141e8542f2559"} Oct 30 18:03:37.571523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:37.570880 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:37.731983 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:37.731950 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:37.907346 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:37.906920 4603 patch_prober.go:29] interesting pod/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 18:03:37.907346 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:37.906977 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" podUID="3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 18:03:38.063534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:38.060151 4603 patch_prober.go:29] interesting pod/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 18:03:38.063534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:38.060214 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" podUID="87c7ffd4-3ef5-4774-a9d9-a29819f2f855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 18:03:38.451918 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:38.451880 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" event={"ID":"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61","Type":"ContainerStarted","Data":"819a52041fe7393a2d1ad32c86cb82e3a2a997c769c71661cc59a981fe4f272d"} Oct 30 18:03:38.453368 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:38.453347 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" Oct 30 18:03:38.751583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:38.750874 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:38.759196 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:38.759106 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:38.807718 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:38.807644 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" podStartSLOduration=3.02433436 podStartE2EDuration="3.807623s" totalImagesPullingTime="783.28864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:38.508769858 +0000 UTC m=+679.224339734" watchObservedRunningTime="2026-10-30 18:03:38.807623 +0000 UTC m=+679.523192879" Oct 30 18:03:38.912524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:38.911898 4603 patch_prober.go:29] interesting pod/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 18:03:38.912524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:38.911967 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" podUID="3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 18:03:39.052775 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:39.052746 4603 patch_prober.go:29] interesting pod/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 18:03:39.052945 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:39.052797 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" podUID="87c7ffd4-3ef5-4774-a9d9-a29819f2f855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 18:03:39.431133 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:39.431003 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:39.918537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:39.918488 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:39.919598 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:39.919579 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:40.052377 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:40.052332 4603 patch_prober.go:29] interesting pod/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 18:03:40.052605 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:40.052394 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" podUID="87c7ffd4-3ef5-4774-a9d9-a29819f2f855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 18:03:40.738306 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:40.738272 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:40.907734 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:40.907702 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:41.056024 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:41.055997 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:41.056387 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:41.056287 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:41.056932 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:41.056914 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:41.065047 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:41.065020 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:41.343720 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:41.343633 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:41.521475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:41.521440 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:41.840039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:41.840007 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p"] Oct 30 18:03:41.840219 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:41.840192 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" podUID="76e15ae2-c331-4780-b383-d99fbd721312" containerName="istio-proxy" containerID="cri-o://9b7b91be8399307fbbd8c6f1832b19f1617ff014dcb3865218e9c7f951ea7589" gracePeriod=30 Oct 30 18:03:41.897740 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:41.897702 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:42.158032 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:42.157952 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:42.171246 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:42.171208 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l"] Oct 30 18:03:42.176079 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:42.176052 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fd500666-qwq5l"] Oct 30 18:03:42.322076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:42.322045 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf"] Oct 30 18:03:42.322296 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:42.322273 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" podUID="3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" containerName="istio-proxy" containerID="cri-o://7c1ffa567f8498caa5a477fa189b5f36d057dc6cead05d14d881a89dbb80a07b" gracePeriod=30 Oct 30 18:03:42.329697 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:42.329667 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:42.492586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:42.486303 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:42.642990 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:42.642949 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:42.799438 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:42.799413 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:42.976967 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:42.976928 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:43.096150 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.096069 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v"] Oct 30 18:03:43.099466 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.099440 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.130768 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.130741 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-6e8fca-openshift-default-dockercfg-sjtwn\"" Oct 30 18:03:43.133659 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.133637 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v"] Oct 30 18:03:43.157974 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.157943 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.158097 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.157987 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.158470 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.158103 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.158470 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.158140 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.158470 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.158179 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.158470 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.158275 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.158470 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.158310 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqnzx\" (UniqueName: \"kubernetes.io/projected/5411300d-c1aa-4397-be3d-0ce869a1c1cb-kube-api-access-qqnzx\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.158470 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.158354 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.158470 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.158426 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.209044 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.209005 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:43.264386 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.264348 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.264726 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.264705 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.264849 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.264832 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.264972 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.264958 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.265094 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.265079 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.265191 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.265178 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqnzx\" (UniqueName: \"kubernetes.io/projected/5411300d-c1aa-4397-be3d-0ce869a1c1cb-kube-api-access-qqnzx\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.265306 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.265295 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.265397 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.265382 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.265549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.265536 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.266086 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.266062 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.267404 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.267381 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.267559 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.265413 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.268002 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.267980 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.268279 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.268228 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.269116 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.269097 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.271479 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.271455 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.296075 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.296041 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqnzx\" (UniqueName: \"kubernetes.io/projected/5411300d-c1aa-4397-be3d-0ce869a1c1cb-kube-api-access-qqnzx\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.296457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.296437 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.380112 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.380042 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:43.411838 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.411806 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a" path="/var/lib/kubelet/pods/f7f2b2a4-8055-4dfe-9598-b5d2fb2e5c8a/volumes" Oct 30 18:03:43.412386 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.412367 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:43.797383 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.797356 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v"] Oct 30 18:03:43.799373 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:03:43.799351 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5411300d_c1aa_4397_be3d_0ce869a1c1cb.slice/crio-1e1a2f77817654ef6ce18656efc4e90c8e00d64f32286ee8f8ec320afd72e736 WatchSource:0}: Error finding container 1e1a2f77817654ef6ce18656efc4e90c8e00d64f32286ee8f8ec320afd72e736: Status 404 returned error can't find the container with id 1e1a2f77817654ef6ce18656efc4e90c8e00d64f32286ee8f8ec320afd72e736 Oct 30 18:03:43.801576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.801543 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:43.801685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.801620 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:43.801685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:43.801663 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:44.502298 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:44.502272 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt"] Oct 30 18:03:44.502699 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:44.502464 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" podUID="e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" containerName="istio-proxy" containerID="cri-o://4ff976d9c9f218b3737c21f0833730cad18723ea25f9ee0a855cee79e90b3317" gracePeriod=30 Oct 30 18:03:44.514562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:44.514531 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" event={"ID":"5411300d-c1aa-4397-be3d-0ce869a1c1cb","Type":"ContainerStarted","Data":"8a93ed842ff210b1ca4d9ffaf50a7eee57466fc23ceed53b565d582dc39d3878"} Oct 30 18:03:44.514683 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:44.514584 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" event={"ID":"5411300d-c1aa-4397-be3d-0ce869a1c1cb","Type":"ContainerStarted","Data":"1e1a2f77817654ef6ce18656efc4e90c8e00d64f32286ee8f8ec320afd72e736"} Oct 30 18:03:44.515947 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:44.515918 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:44.631965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:44.631887 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" podStartSLOduration=1.6318622029999998 podStartE2EDuration="1.631862203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:44.616297215 +0000 UTC m=+685.331867089" watchObservedRunningTime="2026-10-30 18:03:44.631862203 +0000 UTC m=+685.347432078" Oct 30 18:03:44.681967 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:44.681933 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:45.366862 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:45.366827 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:03:45.383635 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:45.383600 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:45.413141 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:45.413111 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:45.418227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:45.418203 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:45.465960 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:45.465940 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" Oct 30 18:03:45.519862 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:45.519834 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:45.520855 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:45.520835 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:03:46.019913 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:46.019862 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:46.368477 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:46.368403 4603 patch_prober.go:29] interesting pod/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:03:46.368477 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:46.368458 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" podUID="e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:03:46.461012 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:46.460977 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:47.083749 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.083726 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:47.086306 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.086280 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:47.125048 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.124986 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-istio-envoy\") pod \"76e15ae2-c331-4780-b383-d99fbd721312\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " Oct 30 18:03:47.125048 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.125021 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-istio-data\") pod \"76e15ae2-c331-4780-b383-d99fbd721312\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " Oct 30 18:03:47.125196 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.125061 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76e15ae2-c331-4780-b383-d99fbd721312-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76e15ae2-c331-4780-b383-d99fbd721312-istiod-ca-cert\") pod \"76e15ae2-c331-4780-b383-d99fbd721312\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " Oct 30 18:03:47.125196 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.125092 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-workload-certs\") pod \"76e15ae2-c331-4780-b383-d99fbd721312\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " Oct 30 18:03:47.125196 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.125117 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76e15ae2-c331-4780-b383-d99fbd721312-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76e15ae2-c331-4780-b383-d99fbd721312-istio-podinfo\") pod \"76e15ae2-c331-4780-b383-d99fbd721312\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " Oct 30 18:03:47.125196 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.125146 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-credential-socket\") pod \"76e15ae2-c331-4780-b383-d99fbd721312\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " Oct 30 18:03:47.125407 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.125289 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76e15ae2-c331-4780-b383-d99fbd721312-istio-token\" (UniqueName: \"kubernetes.io/projected/76e15ae2-c331-4780-b383-d99fbd721312-istio-token\") pod \"76e15ae2-c331-4780-b383-d99fbd721312\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " Oct 30 18:03:47.125407 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.125339 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-workload-socket\") pod \"76e15ae2-c331-4780-b383-d99fbd721312\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " Oct 30 18:03:47.125407 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.125372 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76e15ae2-c331-4780-b383-d99fbd721312-kube-api-access-8cznh\" (UniqueName: \"kubernetes.io/projected/76e15ae2-c331-4780-b383-d99fbd721312-kube-api-access-8cznh\") pod \"76e15ae2-c331-4780-b383-d99fbd721312\" (UID: \"76e15ae2-c331-4780-b383-d99fbd721312\") " Oct 30 18:03:47.125574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.125401 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-istio-data" pod "76e15ae2-c331-4780-b383-d99fbd721312" (UID: "76e15ae2-c331-4780-b383-d99fbd721312"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:47.125574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.125421 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-workload-certs" pod "76e15ae2-c331-4780-b383-d99fbd721312" (UID: "76e15ae2-c331-4780-b383-d99fbd721312"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:47.125682 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.125663 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-workload-socket" pod "76e15ae2-c331-4780-b383-d99fbd721312" (UID: "76e15ae2-c331-4780-b383-d99fbd721312"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:47.125733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.125664 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76e15ae2-c331-4780-b383-d99fbd721312-istiod-ca-cert" pod "76e15ae2-c331-4780-b383-d99fbd721312" (UID: "76e15ae2-c331-4780-b383-d99fbd721312"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:47.125846 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.125819 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-credential-socket" pod "76e15ae2-c331-4780-b383-d99fbd721312" (UID: "76e15ae2-c331-4780-b383-d99fbd721312"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:47.126255 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.126053 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:47.126255 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.126079 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76e15ae2-c331-4780-b383-d99fbd721312-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:47.126255 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.126094 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:47.126255 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.126107 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:47.126255 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.126123 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:47.127468 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.127440 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-istio-envoy" pod "76e15ae2-c331-4780-b383-d99fbd721312" (UID: "76e15ae2-c331-4780-b383-d99fbd721312"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:47.127600 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.127547 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76e15ae2-c331-4780-b383-d99fbd721312-istio-podinfo" pod "76e15ae2-c331-4780-b383-d99fbd721312" (UID: "76e15ae2-c331-4780-b383-d99fbd721312"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:03:47.127600 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.127563 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e15ae2-c331-4780-b383-d99fbd721312-istio-token" pod "76e15ae2-c331-4780-b383-d99fbd721312" (UID: "76e15ae2-c331-4780-b383-d99fbd721312"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:47.127775 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.127758 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e15ae2-c331-4780-b383-d99fbd721312-kube-api-access-8cznh" pod "76e15ae2-c331-4780-b383-d99fbd721312" (UID: "76e15ae2-c331-4780-b383-d99fbd721312"). InnerVolumeSpecName "kube-api-access-8cznh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:47.238564 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.228186 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76e15ae2-c331-4780-b383-d99fbd721312-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:47.238564 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.228233 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76e15ae2-c331-4780-b383-d99fbd721312-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:47.238564 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.228249 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cznh\" (UniqueName: \"kubernetes.io/projected/76e15ae2-c331-4780-b383-d99fbd721312-kube-api-access-8cznh\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:47.238564 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.228263 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76e15ae2-c331-4780-b383-d99fbd721312-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:47.251372 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.251328 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:47.397764 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.397698 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb"] Oct 30 18:03:47.397933 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.397911 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" podUID="87c7ffd4-3ef5-4774-a9d9-a29819f2f855" containerName="istio-proxy" containerID="cri-o://4d1637b1c34ad94d4edc9b953726d8b8710f389c51e7cc642412c7cb0d2e8769" gracePeriod=30 Oct 30 18:03:47.475744 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.475700 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:47.533019 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.532983 4603 generic.go:356] "Generic (PLEG): container finished" podID="76e15ae2-c331-4780-b383-d99fbd721312" containerID="9b7b91be8399307fbbd8c6f1832b19f1617ff014dcb3865218e9c7f951ea7589" exitCode=0 Oct 30 18:03:47.533183 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.533032 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" Oct 30 18:03:47.533183 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.533059 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" event={"ID":"76e15ae2-c331-4780-b383-d99fbd721312","Type":"ContainerDied","Data":"9b7b91be8399307fbbd8c6f1832b19f1617ff014dcb3865218e9c7f951ea7589"} Oct 30 18:03:47.533183 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.533126 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p" event={"ID":"76e15ae2-c331-4780-b383-d99fbd721312","Type":"ContainerDied","Data":"5de2efe09e466b8a01a2fc2121dce5cbde1b6eda4e93f445043ad95790c7b20b"} Oct 30 18:03:47.533183 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.533156 4603 scope.go:122] "RemoveContainer" containerID="9b7b91be8399307fbbd8c6f1832b19f1617ff014dcb3865218e9c7f951ea7589" Oct 30 18:03:47.535551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.535476 4603 generic.go:356] "Generic (PLEG): container finished" podID="3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" containerID="7c1ffa567f8498caa5a477fa189b5f36d057dc6cead05d14d881a89dbb80a07b" exitCode=0 Oct 30 18:03:47.535879 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.535855 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" event={"ID":"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c","Type":"ContainerDied","Data":"7c1ffa567f8498caa5a477fa189b5f36d057dc6cead05d14d881a89dbb80a07b"} Oct 30 18:03:47.544690 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.544675 4603 scope.go:122] "RemoveContainer" containerID="36ccc03ba70c43b3ffc042b6e8ff799d8a322723f9c2475694989a2310dfe8f8" Oct 30 18:03:47.554013 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.553995 4603 scope.go:122] "RemoveContainer" containerID="9b7b91be8399307fbbd8c6f1832b19f1617ff014dcb3865218e9c7f951ea7589" Oct 30 18:03:47.554298 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:03:47.554278 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b7b91be8399307fbbd8c6f1832b19f1617ff014dcb3865218e9c7f951ea7589\": container with ID starting with 9b7b91be8399307fbbd8c6f1832b19f1617ff014dcb3865218e9c7f951ea7589 not found: ID does not exist" containerID="9b7b91be8399307fbbd8c6f1832b19f1617ff014dcb3865218e9c7f951ea7589" Oct 30 18:03:47.554389 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.554304 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7b91be8399307fbbd8c6f1832b19f1617ff014dcb3865218e9c7f951ea7589"} err="failed to get container status \"9b7b91be8399307fbbd8c6f1832b19f1617ff014dcb3865218e9c7f951ea7589\": rpc error: code = NotFound desc = could not find container \"9b7b91be8399307fbbd8c6f1832b19f1617ff014dcb3865218e9c7f951ea7589\": container with ID starting with 9b7b91be8399307fbbd8c6f1832b19f1617ff014dcb3865218e9c7f951ea7589 not found: ID does not exist" Oct 30 18:03:47.554389 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.554319 4603 scope.go:122] "RemoveContainer" containerID="36ccc03ba70c43b3ffc042b6e8ff799d8a322723f9c2475694989a2310dfe8f8" Oct 30 18:03:47.554644 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:03:47.554562 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36ccc03ba70c43b3ffc042b6e8ff799d8a322723f9c2475694989a2310dfe8f8\": container with ID starting with 36ccc03ba70c43b3ffc042b6e8ff799d8a322723f9c2475694989a2310dfe8f8 not found: ID does not exist" containerID="36ccc03ba70c43b3ffc042b6e8ff799d8a322723f9c2475694989a2310dfe8f8" Oct 30 18:03:47.554644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.554586 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36ccc03ba70c43b3ffc042b6e8ff799d8a322723f9c2475694989a2310dfe8f8"} err="failed to get container status \"36ccc03ba70c43b3ffc042b6e8ff799d8a322723f9c2475694989a2310dfe8f8\": rpc error: code = NotFound desc = could not find container \"36ccc03ba70c43b3ffc042b6e8ff799d8a322723f9c2475694989a2310dfe8f8\": container with ID starting with 36ccc03ba70c43b3ffc042b6e8ff799d8a322723f9c2475694989a2310dfe8f8 not found: ID does not exist" Oct 30 18:03:47.554644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.554599 4603 scope.go:122] "RemoveContainer" containerID="07f0d6b90e91fa43badf59c24d209b41e6553e3888dd31e8ba45951cc2df638b" Oct 30 18:03:47.648424 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.648294 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p"] Oct 30 18:03:47.652780 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.651005 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ce05c7ef-openshift-default-b9bb988f-zsg2p"] Oct 30 18:03:47.657870 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.657836 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:47.688174 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.688146 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:47.743720 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.743696 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-kube-api-access-h2gnz\" (UniqueName: \"kubernetes.io/projected/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-kube-api-access-h2gnz\") pod \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " Oct 30 18:03:47.743883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.743748 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-podinfo\") pod \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " Oct 30 18:03:47.743883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.743771 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-workload-certs\") pod \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " Oct 30 18:03:47.744001 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.743920 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-workload-socket\") pod \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " Oct 30 18:03:47.744001 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.743967 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-envoy\") pod \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " Oct 30 18:03:47.744113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.744016 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-data\") pod \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " Oct 30 18:03:47.744113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.744064 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-credential-socket\") pod \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " Oct 30 18:03:47.744113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.744067 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-workload-certs" pod "3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" (UID: "3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:47.744113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.744102 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istiod-ca-cert\") pod \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " Oct 30 18:03:47.744321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.744157 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-token\" (UniqueName: \"kubernetes.io/projected/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-token\") pod \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\" (UID: \"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c\") " Oct 30 18:03:47.744321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.744259 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-data" pod "3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" (UID: "3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:47.744422 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.744349 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-workload-socket" pod "3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" (UID: "3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:47.744473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.744449 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-credential-socket" pod "3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" (UID: "3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:47.744803 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.744746 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istiod-ca-cert" pod "3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" (UID: "3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:47.745343 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.745319 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:47.745463 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.745344 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:47.745463 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.745373 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:47.745463 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.745386 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:47.745463 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.745398 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:47.746490 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.746470 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-envoy" pod "3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" (UID: "3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:47.746718 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.746697 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-kube-api-access-h2gnz" pod "3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" (UID: "3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c"). InnerVolumeSpecName "kube-api-access-h2gnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:47.746718 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.746696 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-podinfo" pod "3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" (UID: "3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:03:47.746983 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.746963 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-token" pod "3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" (UID: "3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:47.845966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.845929 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2gnz\" (UniqueName: \"kubernetes.io/projected/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-kube-api-access-h2gnz\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:47.845966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.845961 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:47.845966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.845972 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:47.846217 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:47.845981 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:48.262032 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.261993 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:48.543412 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.543386 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" Oct 30 18:03:48.543412 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.543387 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf" event={"ID":"3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c","Type":"ContainerDied","Data":"98e05d212eb7f120f5019088d71f35cc5ea7d0ee87e2c9fe5d43d88184072aca"} Oct 30 18:03:48.543671 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.543463 4603 scope.go:122] "RemoveContainer" containerID="7c1ffa567f8498caa5a477fa189b5f36d057dc6cead05d14d881a89dbb80a07b" Oct 30 18:03:48.621763 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.621732 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf"] Oct 30 18:03:48.630686 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.630656 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-fd500666-gw-openshift-default-b5867b669-pg9kf"] Oct 30 18:03:48.754644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.754617 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp"] Oct 30 18:03:48.755216 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.755200 4603 cpu_manager.go:414] "containerMap: removing container" podUID="3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" containerName="istio-proxy" Oct 30 18:03:48.755277 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.755219 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" containerName="istio-proxy" Oct 30 18:03:48.755277 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.755241 4603 cpu_manager.go:414] "containerMap: removing container" podUID="3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" containerName="istio-proxy" Oct 30 18:03:48.755277 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.755249 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" containerName="istio-proxy" Oct 30 18:03:48.755277 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.755262 4603 cpu_manager.go:414] "containerMap: removing container" podUID="76e15ae2-c331-4780-b383-d99fbd721312" containerName="istio-proxy" Oct 30 18:03:48.755277 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.755269 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76e15ae2-c331-4780-b383-d99fbd721312" containerName="istio-proxy" Oct 30 18:03:48.755552 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.755289 4603 cpu_manager.go:414] "containerMap: removing container" podUID="76e15ae2-c331-4780-b383-d99fbd721312" containerName="istio-proxy" Oct 30 18:03:48.755552 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.755296 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76e15ae2-c331-4780-b383-d99fbd721312" containerName="istio-proxy" Oct 30 18:03:48.755552 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.755373 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" containerName="istio-proxy" Oct 30 18:03:48.755552 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.755380 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" containerName="istio-proxy" Oct 30 18:03:48.755552 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.755387 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="76e15ae2-c331-4780-b383-d99fbd721312" containerName="istio-proxy" Oct 30 18:03:48.755552 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.755393 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="76e15ae2-c331-4780-b383-d99fbd721312" containerName="istio-proxy" Oct 30 18:03:48.759087 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.759069 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.783487 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.783467 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b424a5-openshift-default-dockercfg-gfhfl\"" Oct 30 18:03:48.783633 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.783618 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp"] Oct 30 18:03:48.845400 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.845315 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:48.869705 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.869667 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.870142 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.870113 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.870274 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.870174 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.870274 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.870249 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.870393 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.870337 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.870445 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.870395 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.870445 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.870414 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.870587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.870550 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms98v\" (UniqueName: \"kubernetes.io/projected/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-kube-api-access-ms98v\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.870646 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.870605 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.972005 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.971975 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.972005 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.972020 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.972218 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.972035 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.972218 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.972078 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.972218 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.972099 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.972218 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.972122 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.972218 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.972137 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.972218 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.972185 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms98v\" (UniqueName: \"kubernetes.io/projected/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-kube-api-access-ms98v\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.972547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.972247 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.972547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.972474 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.972547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.972491 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.972719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.972674 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.973067 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.973043 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.973159 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.973146 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.974625 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.974602 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:48.974897 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:48.974877 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:49.017326 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:49.017269 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:49.027932 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:49.021206 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms98v\" (UniqueName: \"kubernetes.io/projected/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-kube-api-access-ms98v\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:49.070055 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:49.070013 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:49.070799 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:49.070780 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:49.326873 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:49.326836 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:49.419964 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:49.419875 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c" path="/var/lib/kubelet/pods/3ae1fe54-cb96-4cf9-9d30-5d5d67f3b25c/volumes" Oct 30 18:03:49.420530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:49.420479 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76e15ae2-c331-4780-b383-d99fbd721312" path="/var/lib/kubelet/pods/76e15ae2-c331-4780-b383-d99fbd721312/volumes" Oct 30 18:03:49.422272 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:49.422250 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp"] Oct 30 18:03:49.424845 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:03:49.424822 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a63a4e_bf67_43a5_98b0_f38ca2cb7685.slice/crio-90b8dd53bd51c2d39b0c3eea01177e89a69fcdd6c2da68bd9c992021cc3cc4ab WatchSource:0}: Error finding container 90b8dd53bd51c2d39b0c3eea01177e89a69fcdd6c2da68bd9c992021cc3cc4ab: Status 404 returned error can't find the container with id 90b8dd53bd51c2d39b0c3eea01177e89a69fcdd6c2da68bd9c992021cc3cc4ab Oct 30 18:03:49.426947 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:49.426922 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:49.427008 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:49.426984 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:49.427041 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:49.427015 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:49.526153 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:49.526093 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:49.562539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:49.559387 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" event={"ID":"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685","Type":"ContainerStarted","Data":"53a9ec2a86a85da94835ae4ea315d8040fd15baad1f9e73f353dd19c01b217a3"} Oct 30 18:03:49.562539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:49.559514 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" event={"ID":"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685","Type":"ContainerStarted","Data":"90b8dd53bd51c2d39b0c3eea01177e89a69fcdd6c2da68bd9c992021cc3cc4ab"} Oct 30 18:03:49.645549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:49.617709 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" podStartSLOduration=1.6176858680000001 podStartE2EDuration="1.617685868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:49.616200655 +0000 UTC m=+690.331770553" watchObservedRunningTime="2026-10-30 18:03:49.617685868 +0000 UTC m=+690.333255743" Oct 30 18:03:49.720859 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:49.720788 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:49.929599 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:49.929574 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:50.056746 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.056712 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-envoy\") pod \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " Oct 30 18:03:50.056903 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.056752 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-data\") pod \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " Oct 30 18:03:50.056903 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.056795 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-kube-api-access-wndkp\" (UniqueName: \"kubernetes.io/projected/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-kube-api-access-wndkp\") pod \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " Oct 30 18:03:50.056903 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.056864 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istiod-ca-cert\") pod \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " Oct 30 18:03:50.057085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.056911 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-podinfo\") pod \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " Oct 30 18:03:50.057085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.056963 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-credential-socket\") pod \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " Oct 30 18:03:50.057085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.057015 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-token\" (UniqueName: \"kubernetes.io/projected/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-token\") pod \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " Oct 30 18:03:50.057220 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.057086 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-data" pod "e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" (UID: "e2d00243-a6a6-438f-ab25-9ae2ac5e75bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:50.057220 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.057100 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-workload-certs\") pod \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " Oct 30 18:03:50.057328 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.057221 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-workload-socket\") pod \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\" (UID: \"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd\") " Oct 30 18:03:50.057328 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.057265 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-credential-socket" pod "e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" (UID: "e2d00243-a6a6-438f-ab25-9ae2ac5e75bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:50.057419 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.057346 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-workload-certs" pod "e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" (UID: "e2d00243-a6a6-438f-ab25-9ae2ac5e75bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:50.057419 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.057364 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istiod-ca-cert" pod "e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" (UID: "e2d00243-a6a6-438f-ab25-9ae2ac5e75bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:50.057585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.057569 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-workload-socket" pod "e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" (UID: "e2d00243-a6a6-438f-ab25-9ae2ac5e75bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:50.057876 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.057849 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:50.057955 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.057878 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:50.057955 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.057894 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:50.057955 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.057908 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:50.057955 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.057922 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:50.059078 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.059054 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-envoy" pod "e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" (UID: "e2d00243-a6a6-438f-ab25-9ae2ac5e75bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:50.059263 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.059239 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-kube-api-access-wndkp" pod "e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" (UID: "e2d00243-a6a6-438f-ab25-9ae2ac5e75bd"). InnerVolumeSpecName "kube-api-access-wndkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:50.059380 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.059363 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-podinfo" pod "e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" (UID: "e2d00243-a6a6-438f-ab25-9ae2ac5e75bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:03:50.059553 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.059539 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-token" pod "e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" (UID: "e2d00243-a6a6-438f-ab25-9ae2ac5e75bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:50.071723 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.071701 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:50.076217 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.076201 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:50.159187 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.159160 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:50.159187 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.159184 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wndkp\" (UniqueName: \"kubernetes.io/projected/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-kube-api-access-wndkp\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:50.159351 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.159196 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:50.159351 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.159205 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:50.570278 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.570244 4603 generic.go:356] "Generic (PLEG): container finished" podID="e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" containerID="4ff976d9c9f218b3737c21f0833730cad18723ea25f9ee0a855cee79e90b3317" exitCode=0 Oct 30 18:03:50.570769 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.570298 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" Oct 30 18:03:50.570769 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.570335 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" event={"ID":"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd","Type":"ContainerDied","Data":"4ff976d9c9f218b3737c21f0833730cad18723ea25f9ee0a855cee79e90b3317"} Oct 30 18:03:50.570769 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.570398 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt" event={"ID":"e2d00243-a6a6-438f-ab25-9ae2ac5e75bd","Type":"ContainerDied","Data":"6db5471efd2999fe92af49354b4187ebf87ceccad2a3ba6719ea9fde13818f96"} Oct 30 18:03:50.570769 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.570435 4603 scope.go:122] "RemoveContainer" containerID="4ff976d9c9f218b3737c21f0833730cad18723ea25f9ee0a855cee79e90b3317" Oct 30 18:03:50.570943 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.570900 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:50.571777 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.571760 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:03:50.629527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.627349 4603 scope.go:122] "RemoveContainer" containerID="452b45a9d92b4b5af4aa38811bb5bd9d8973cd2a79b3e94b4dcd169ac25c0bfa" Oct 30 18:03:50.638309 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.638290 4603 scope.go:122] "RemoveContainer" containerID="4ff976d9c9f218b3737c21f0833730cad18723ea25f9ee0a855cee79e90b3317" Oct 30 18:03:50.638652 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:03:50.638630 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ff976d9c9f218b3737c21f0833730cad18723ea25f9ee0a855cee79e90b3317\": container with ID starting with 4ff976d9c9f218b3737c21f0833730cad18723ea25f9ee0a855cee79e90b3317 not found: ID does not exist" containerID="4ff976d9c9f218b3737c21f0833730cad18723ea25f9ee0a855cee79e90b3317" Oct 30 18:03:50.638717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.638662 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ff976d9c9f218b3737c21f0833730cad18723ea25f9ee0a855cee79e90b3317"} err="failed to get container status \"4ff976d9c9f218b3737c21f0833730cad18723ea25f9ee0a855cee79e90b3317\": rpc error: code = NotFound desc = could not find container \"4ff976d9c9f218b3737c21f0833730cad18723ea25f9ee0a855cee79e90b3317\": container with ID starting with 4ff976d9c9f218b3737c21f0833730cad18723ea25f9ee0a855cee79e90b3317 not found: ID does not exist" Oct 30 18:03:50.638717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.638681 4603 scope.go:122] "RemoveContainer" containerID="452b45a9d92b4b5af4aa38811bb5bd9d8973cd2a79b3e94b4dcd169ac25c0bfa" Oct 30 18:03:50.638928 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:03:50.638907 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"452b45a9d92b4b5af4aa38811bb5bd9d8973cd2a79b3e94b4dcd169ac25c0bfa\": container with ID starting with 452b45a9d92b4b5af4aa38811bb5bd9d8973cd2a79b3e94b4dcd169ac25c0bfa not found: ID does not exist" containerID="452b45a9d92b4b5af4aa38811bb5bd9d8973cd2a79b3e94b4dcd169ac25c0bfa" Oct 30 18:03:50.638983 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.638936 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452b45a9d92b4b5af4aa38811bb5bd9d8973cd2a79b3e94b4dcd169ac25c0bfa"} err="failed to get container status \"452b45a9d92b4b5af4aa38811bb5bd9d8973cd2a79b3e94b4dcd169ac25c0bfa\": rpc error: code = NotFound desc = could not find container \"452b45a9d92b4b5af4aa38811bb5bd9d8973cd2a79b3e94b4dcd169ac25c0bfa\": container with ID starting with 452b45a9d92b4b5af4aa38811bb5bd9d8973cd2a79b3e94b4dcd169ac25c0bfa not found: ID does not exist" Oct 30 18:03:50.687725 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.687698 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt"] Oct 30 18:03:50.714755 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.701797 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a8c61e05-openshift-default-58cbc746dc-mpsdt"] Oct 30 18:03:50.949339 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:50.949253 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:51.412538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:51.411820 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:51.414963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:51.414932 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" path="/var/lib/kubelet/pods/e2d00243-a6a6-438f-ab25-9ae2ac5e75bd/volumes" Oct 30 18:03:51.615080 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:51.615044 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:51.769057 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:51.768980 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:51.940973 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:03:51.940930 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:03:52.245296 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.245256 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:03:52.587877 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.587845 4603 generic.go:356] "Generic (PLEG): container finished" podID="87c7ffd4-3ef5-4774-a9d9-a29819f2f855" containerID="4d1637b1c34ad94d4edc9b953726d8b8710f389c51e7cc642412c7cb0d2e8769" exitCode=0 Oct 30 18:03:52.588032 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.587914 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" event={"ID":"87c7ffd4-3ef5-4774-a9d9-a29819f2f855","Type":"ContainerDied","Data":"4d1637b1c34ad94d4edc9b953726d8b8710f389c51e7cc642412c7cb0d2e8769"} Oct 30 18:03:52.588032 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.587977 4603 scope.go:122] "RemoveContainer" containerID="26ffb791a2d79b2211236c2471499bbb8aedb696d78ddd36bd710e0f0b0d2061" Oct 30 18:03:52.743629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.743596 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:52.809529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.804271 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-workload-socket\") pod \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " Oct 30 18:03:52.809529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.804328 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-kube-api-access-zdwcv\" (UniqueName: \"kubernetes.io/projected/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-kube-api-access-zdwcv\") pod \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " Oct 30 18:03:52.809529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.804380 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-credential-socket\") pod \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " Oct 30 18:03:52.809529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.804425 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-workload-certs\") pod \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " Oct 30 18:03:52.809529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.804457 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-podinfo\") pod \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " Oct 30 18:03:52.809529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.804592 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istiod-ca-cert\") pod \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " Oct 30 18:03:52.809529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.804628 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-envoy\") pod \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " Oct 30 18:03:52.809529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.804666 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-token\" (UniqueName: \"kubernetes.io/projected/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-token\") pod \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " Oct 30 18:03:52.809529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.804696 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-data\") pod \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\" (UID: \"87c7ffd4-3ef5-4774-a9d9-a29819f2f855\") " Oct 30 18:03:52.809529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.805248 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-data" pod "87c7ffd4-3ef5-4774-a9d9-a29819f2f855" (UID: "87c7ffd4-3ef5-4774-a9d9-a29819f2f855"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:52.809529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.805364 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-workload-socket" pod "87c7ffd4-3ef5-4774-a9d9-a29819f2f855" (UID: "87c7ffd4-3ef5-4774-a9d9-a29819f2f855"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:52.809529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.805633 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-credential-socket" pod "87c7ffd4-3ef5-4774-a9d9-a29819f2f855" (UID: "87c7ffd4-3ef5-4774-a9d9-a29819f2f855"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:52.809529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.805660 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istiod-ca-cert" pod "87c7ffd4-3ef5-4774-a9d9-a29819f2f855" (UID: "87c7ffd4-3ef5-4774-a9d9-a29819f2f855"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:52.809529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.805937 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-workload-certs" pod "87c7ffd4-3ef5-4774-a9d9-a29819f2f855" (UID: "87c7ffd4-3ef5-4774-a9d9-a29819f2f855"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:52.816107 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.816060 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-podinfo" pod "87c7ffd4-3ef5-4774-a9d9-a29819f2f855" (UID: "87c7ffd4-3ef5-4774-a9d9-a29819f2f855"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:03:52.816717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.816685 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-kube-api-access-zdwcv" pod "87c7ffd4-3ef5-4774-a9d9-a29819f2f855" (UID: "87c7ffd4-3ef5-4774-a9d9-a29819f2f855"). InnerVolumeSpecName "kube-api-access-zdwcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:52.816832 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.816816 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-envoy" pod "87c7ffd4-3ef5-4774-a9d9-a29819f2f855" (UID: "87c7ffd4-3ef5-4774-a9d9-a29819f2f855"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:52.816892 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.816864 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-token" pod "87c7ffd4-3ef5-4774-a9d9-a29819f2f855" (UID: "87c7ffd4-3ef5-4774-a9d9-a29819f2f855"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:52.913527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.910170 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:52.913527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.910205 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:52.913527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.910220 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:52.913527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.910233 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:52.913527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.910246 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:52.913527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.910261 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdwcv\" (UniqueName: \"kubernetes.io/projected/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-kube-api-access-zdwcv\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:52.913527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.910276 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:52.913527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.910289 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:52.913527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:52.910302 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c7ffd4-3ef5-4774-a9d9-a29819f2f855-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:03:53.221569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.221459 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:53.515089 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.514984 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:53.594116 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.594084 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" event={"ID":"87c7ffd4-3ef5-4774-a9d9-a29819f2f855","Type":"ContainerDied","Data":"fc553b1eecefda51692c6fd732154cb37309d65bc76a64a739c83911bcdceb51"} Oct 30 18:03:53.594116 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.594108 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb" Oct 30 18:03:53.594339 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.594152 4603 scope.go:122] "RemoveContainer" containerID="4d1637b1c34ad94d4edc9b953726d8b8710f389c51e7cc642412c7cb0d2e8769" Oct 30 18:03:53.685117 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.685091 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb"] Oct 30 18:03:53.707872 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.707836 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-5e0da928-openshift-default-6587d8f4df-5mxzb"] Oct 30 18:03:53.717975 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.717954 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn"] Oct 30 18:03:53.718321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.718307 4603 cpu_manager.go:414] "containerMap: removing container" podUID="e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" containerName="istio-proxy" Oct 30 18:03:53.718373 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.718322 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" containerName="istio-proxy" Oct 30 18:03:53.718373 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.718340 4603 cpu_manager.go:414] "containerMap: removing container" podUID="87c7ffd4-3ef5-4774-a9d9-a29819f2f855" containerName="istio-proxy" Oct 30 18:03:53.718373 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.718346 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c7ffd4-3ef5-4774-a9d9-a29819f2f855" containerName="istio-proxy" Oct 30 18:03:53.718373 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.718355 4603 cpu_manager.go:414] "containerMap: removing container" podUID="87c7ffd4-3ef5-4774-a9d9-a29819f2f855" containerName="istio-proxy" Oct 30 18:03:53.718373 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.718362 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c7ffd4-3ef5-4774-a9d9-a29819f2f855" containerName="istio-proxy" Oct 30 18:03:53.718562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.718435 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="87c7ffd4-3ef5-4774-a9d9-a29819f2f855" containerName="istio-proxy" Oct 30 18:03:53.718562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.718442 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="87c7ffd4-3ef5-4774-a9d9-a29819f2f855" containerName="istio-proxy" Oct 30 18:03:53.718562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.718453 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" containerName="istio-proxy" Oct 30 18:03:53.718562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.718459 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" containerName="istio-proxy" Oct 30 18:03:53.722622 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.722606 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.753096 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.753074 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-3cc458-openshift-default-dockercfg-ld6nz\"" Oct 30 18:03:53.757950 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.757931 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn"] Oct 30 18:03:53.810291 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.810263 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:53.821367 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.821342 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e0b280-2b65-4fad-8344-67049190d108-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.821465 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.821376 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e0b280-2b65-4fad-8344-67049190d108-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.821465 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.821398 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.821465 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.821415 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.821648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.821618 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e0b280-2b65-4fad-8344-67049190d108-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.821728 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.821713 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.821785 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.821756 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.821834 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.821790 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk8wl\" (UniqueName: \"kubernetes.io/projected/22e0b280-2b65-4fad-8344-67049190d108-kube-api-access-rk8wl\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.821904 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.821883 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.923066 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.923030 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e0b280-2b65-4fad-8344-67049190d108-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.923270 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.923096 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e0b280-2b65-4fad-8344-67049190d108-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.923270 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.923128 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.923270 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.923155 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.923270 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.923197 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e0b280-2b65-4fad-8344-67049190d108-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.923472 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.923274 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.923472 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.923303 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.923472 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.923339 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk8wl\" (UniqueName: \"kubernetes.io/projected/22e0b280-2b65-4fad-8344-67049190d108-kube-api-access-rk8wl\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.923472 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.923381 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.923850 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.923823 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.925020 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.924532 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.926081 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.926021 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.931776 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.931751 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.932373 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.932340 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e0b280-2b65-4fad-8344-67049190d108-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.933987 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.933938 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.935224 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.935204 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e0b280-2b65-4fad-8344-67049190d108-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.953166 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.953108 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e0b280-2b65-4fad-8344-67049190d108-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.955332 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.955303 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk8wl\" (UniqueName: \"kubernetes.io/projected/22e0b280-2b65-4fad-8344-67049190d108-kube-api-access-rk8wl\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:53.999085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:53.999053 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:54.034851 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:54.034826 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:54.355459 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:54.355433 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn"] Oct 30 18:03:54.358308 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:03:54.358269 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e0b280_2b65_4fad_8344_67049190d108.slice/crio-a70932a0e69be2689d6c7f15f49d23a7e99995820976c73873f5b54704807349 WatchSource:0}: Error finding container a70932a0e69be2689d6c7f15f49d23a7e99995820976c73873f5b54704807349: Status 404 returned error can't find the container with id a70932a0e69be2689d6c7f15f49d23a7e99995820976c73873f5b54704807349 Oct 30 18:03:54.360912 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:54.360878 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:54.361040 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:54.360943 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:54.361040 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:54.360973 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:54.603439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:54.602763 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" event={"ID":"22e0b280-2b65-4fad-8344-67049190d108","Type":"ContainerStarted","Data":"16d8aa22cecbcf7f4440eecd90d2bc6e688028c1e367f0d43f4f2c01b4726a68"} Oct 30 18:03:54.603439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:54.602866 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" event={"ID":"22e0b280-2b65-4fad-8344-67049190d108","Type":"ContainerStarted","Data":"a70932a0e69be2689d6c7f15f49d23a7e99995820976c73873f5b54704807349"} Oct 30 18:03:54.651147 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:54.651039 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" podStartSLOduration=1.651020603 podStartE2EDuration="1.651020603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:54.648130902 +0000 UTC m=+695.363700770" watchObservedRunningTime="2026-10-30 18:03:54.651020603 +0000 UTC m=+695.366590479" Oct 30 18:03:55.041566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:55.041463 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:55.075207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:55.075166 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:55.241258 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:55.241132 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:03:55.409033 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:55.408965 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87c7ffd4-3ef5-4774-a9d9-a29819f2f855" path="/var/lib/kubelet/pods/87c7ffd4-3ef5-4774-a9d9-a29819f2f855/volumes" Oct 30 18:03:55.526418 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:55.526383 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:55.607744 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:55.607704 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:55.608638 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:55.608617 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:03:55.691012 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:55.690917 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:56.299763 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:56.299712 4603 patch_prober.go:29] interesting pod/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:03:56.300282 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:56.299801 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" podUID="e355edcc-648d-492b-9665-2107f84afb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:03:57.486157 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:57.484186 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:57.621332 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:57.621297 4603 generic.go:356] "Generic (PLEG): container finished" podID="e355edcc-648d-492b-9665-2107f84afb63" containerID="2258b6a8808bef06a850862f288d2b4769736952d4d1efe9f87a65145af06b69" exitCode=0 Oct 30 18:03:57.621561 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:57.621373 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" event={"ID":"e355edcc-648d-492b-9665-2107f84afb63","Type":"ContainerDied","Data":"2258b6a8808bef06a850862f288d2b4769736952d4d1efe9f87a65145af06b69"} Oct 30 18:03:57.621915 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:57.621897 4603 scope.go:122] "RemoveContainer" containerID="2258b6a8808bef06a850862f288d2b4769736952d4d1efe9f87a65145af06b69" Oct 30 18:03:57.623037 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:57.623013 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:57.623146 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:57.623088 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:57.623210 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:57.623150 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:03:58.616528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:58.615208 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:03:58.673173 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:58.673143 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" event={"ID":"e355edcc-648d-492b-9665-2107f84afb63","Type":"ContainerStarted","Data":"d516d859312996808ec5ad34b36d545a649b85d0ac55ed9fc249e3f7221846d9"} Oct 30 18:03:59.006051 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:59.005956 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:03:59.006412 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:59.006388 4603 patch_prober.go:29] interesting pod/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 18:03:59.011689 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:59.011653 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" podUID="e355edcc-648d-492b-9665-2107f84afb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 18:03:59.167983 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:03:59.167937 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:00.024641 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:00.024606 4603 patch_prober.go:29] interesting pod/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 18:04:00.025287 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:00.025251 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" podUID="e355edcc-648d-492b-9665-2107f84afb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 18:04:01.003636 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:01.003596 4603 patch_prober.go:29] interesting pod/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 18:04:01.003906 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:01.003875 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" podUID="e355edcc-648d-492b-9665-2107f84afb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 18:04:01.255363 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:01.255271 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:01.439372 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:01.439216 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:01.996864 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:01.996827 4603 patch_prober.go:29] interesting pod/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 18:04:02.006525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:02.002575 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" podUID="e355edcc-648d-492b-9665-2107f84afb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 18:04:02.040634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:02.040601 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:02.748755 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:02.748669 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:02.961245 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:02.961209 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:03.000576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:02.997818 4603 patch_prober.go:29] interesting pod/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 18:04:03.000576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:02.997877 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" podUID="e355edcc-648d-492b-9665-2107f84afb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 18:04:03.569802 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:03.569765 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:03.997435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:03.997353 4603 patch_prober.go:29] interesting pod/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 18:04:03.997925 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:03.997449 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" podUID="e355edcc-648d-492b-9665-2107f84afb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 18:04:04.047348 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:04.047316 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:04.186841 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:04.186808 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:05.000613 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:05.000583 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:04:05.001015 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:05.000982 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:04:05.001594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:05.001578 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:04:05.364135 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:05.364101 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:06.750222 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:06.750147 4603 generic.go:356] "Generic (PLEG): container finished" podID="5411300d-c1aa-4397-be3d-0ce869a1c1cb" containerID="8a93ed842ff210b1ca4d9ffaf50a7eee57466fc23ceed53b565d582dc39d3878" exitCode=0 Oct 30 18:04:06.750627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:06.750223 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" event={"ID":"5411300d-c1aa-4397-be3d-0ce869a1c1cb","Type":"ContainerDied","Data":"8a93ed842ff210b1ca4d9ffaf50a7eee57466fc23ceed53b565d582dc39d3878"} Oct 30 18:04:06.750672 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:06.750649 4603 scope.go:122] "RemoveContainer" containerID="8a93ed842ff210b1ca4d9ffaf50a7eee57466fc23ceed53b565d582dc39d3878" Oct 30 18:04:06.751819 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:06.751789 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:04:06.751897 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:06.751857 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:04:06.751897 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:06.751888 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:04:07.197399 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:07.197311 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:07.353533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:07.353481 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:07.756784 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:07.756749 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" event={"ID":"5411300d-c1aa-4397-be3d-0ce869a1c1cb","Type":"ContainerStarted","Data":"d3651a6683955db5bd115cb4b16bb4ff672c158d35fd8f25cf04586a82ad9e4b"} Oct 30 18:04:08.413719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:08.413681 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:04:08.413919 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:08.413856 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:04:08.413996 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:08.413910 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" podUID="5411300d-c1aa-4397-be3d-0ce869a1c1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 18:04:09.019365 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:09.019333 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:09.217897 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:09.217865 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:09.377134 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:09.377050 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:09.414979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:09.414948 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:04:09.415242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:09.415217 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" podUID="5411300d-c1aa-4397-be3d-0ce869a1c1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 18:04:09.544329 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:09.544287 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:10.413913 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:10.413881 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:04:10.414400 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:10.413944 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" podUID="5411300d-c1aa-4397-be3d-0ce869a1c1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 18:04:10.921922 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:10.920572 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz"] Oct 30 18:04:10.921922 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:10.920856 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" podUID="fedfff5d-d5b3-4a6f-b126-34fb9c52ac61" containerName="maas-api" containerID="cri-o://819a52041fe7393a2d1ad32c86cb82e3a2a997c769c71661cc59a981fe4f272d" gracePeriod=30 Oct 30 18:04:11.007046 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.007015 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:11.175760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.175697 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" Oct 30 18:04:11.325257 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.325217 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-maas-api-tls\") pod \"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61\" (UID: \"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61\") " Oct 30 18:04:11.325446 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.325332 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-openshift-service-ca\") pod \"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61\" (UID: \"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61\") " Oct 30 18:04:11.325446 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.325379 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-metrics-certs\") pod \"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61\" (UID: \"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61\") " Oct 30 18:04:11.325446 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.325412 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-kube-api-access-z9v2q\" (UniqueName: \"kubernetes.io/projected/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-kube-api-access-z9v2q\") pod \"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61\" (UID: \"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61\") " Oct 30 18:04:11.325826 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.325790 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-openshift-service-ca" pod "fedfff5d-d5b3-4a6f-b126-34fb9c52ac61" (UID: "fedfff5d-d5b3-4a6f-b126-34fb9c52ac61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:04:11.327737 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.327703 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-metrics-certs" pod "fedfff5d-d5b3-4a6f-b126-34fb9c52ac61" (UID: "fedfff5d-d5b3-4a6f-b126-34fb9c52ac61"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:04:11.327826 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.327710 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-maas-api-tls" pod "fedfff5d-d5b3-4a6f-b126-34fb9c52ac61" (UID: "fedfff5d-d5b3-4a6f-b126-34fb9c52ac61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:04:11.327826 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.327739 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-kube-api-access-z9v2q" pod "fedfff5d-d5b3-4a6f-b126-34fb9c52ac61" (UID: "fedfff5d-d5b3-4a6f-b126-34fb9c52ac61"). InnerVolumeSpecName "kube-api-access-z9v2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:11.423524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.423312 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:04:11.423524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.423371 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" podUID="5411300d-c1aa-4397-be3d-0ce869a1c1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 18:04:11.432231 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.432154 4603 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-openshift-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:04:11.432437 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.432419 4603 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:04:11.432563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.432547 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9v2q\" (UniqueName: \"kubernetes.io/projected/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-kube-api-access-z9v2q\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:04:11.432664 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.432652 4603 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61-maas-api-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:04:11.541359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.541322 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:11.708130 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.708038 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:11.778318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.778284 4603 generic.go:356] "Generic (PLEG): container finished" podID="fedfff5d-d5b3-4a6f-b126-34fb9c52ac61" containerID="819a52041fe7393a2d1ad32c86cb82e3a2a997c769c71661cc59a981fe4f272d" exitCode=0 Oct 30 18:04:11.778518 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.778336 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" Oct 30 18:04:11.778518 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.778358 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" event={"ID":"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61","Type":"ContainerDied","Data":"819a52041fe7393a2d1ad32c86cb82e3a2a997c769c71661cc59a981fe4f272d"} Oct 30 18:04:11.778518 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.778400 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz" event={"ID":"fedfff5d-d5b3-4a6f-b126-34fb9c52ac61","Type":"ContainerDied","Data":"ccee212f0157977e8ac97e8cbad9ff12718b5c6ce29be25d60f141e8542f2559"} Oct 30 18:04:11.778518 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.778420 4603 scope.go:122] "RemoveContainer" containerID="819a52041fe7393a2d1ad32c86cb82e3a2a997c769c71661cc59a981fe4f272d" Oct 30 18:04:11.787294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.787275 4603 scope.go:122] "RemoveContainer" containerID="819a52041fe7393a2d1ad32c86cb82e3a2a997c769c71661cc59a981fe4f272d" Oct 30 18:04:11.787568 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:04:11.787547 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"819a52041fe7393a2d1ad32c86cb82e3a2a997c769c71661cc59a981fe4f272d\": container with ID starting with 819a52041fe7393a2d1ad32c86cb82e3a2a997c769c71661cc59a981fe4f272d not found: ID does not exist" containerID="819a52041fe7393a2d1ad32c86cb82e3a2a997c769c71661cc59a981fe4f272d" Oct 30 18:04:11.787659 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.787579 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"819a52041fe7393a2d1ad32c86cb82e3a2a997c769c71661cc59a981fe4f272d"} err="failed to get container status \"819a52041fe7393a2d1ad32c86cb82e3a2a997c769c71661cc59a981fe4f272d\": rpc error: code = NotFound desc = could not find container \"819a52041fe7393a2d1ad32c86cb82e3a2a997c769c71661cc59a981fe4f272d\": container with ID starting with 819a52041fe7393a2d1ad32c86cb82e3a2a997c769c71661cc59a981fe4f272d not found: ID does not exist" Oct 30 18:04:11.833211 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.833179 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz"] Oct 30 18:04:11.837516 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:11.837485 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a9a40825-567c6c7c44-lxjgz"] Oct 30 18:04:12.413863 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:12.413823 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:04:12.414072 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:12.413890 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" podUID="5411300d-c1aa-4397-be3d-0ce869a1c1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 18:04:12.739583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:12.739469 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:13.408386 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:13.408353 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fedfff5d-d5b3-4a6f-b126-34fb9c52ac61" path="/var/lib/kubelet/pods/fedfff5d-d5b3-4a6f-b126-34fb9c52ac61/volumes" Oct 30 18:04:13.417690 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:13.417668 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:04:13.418028 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:13.418009 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:04:13.418739 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:13.418720 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:04:13.498939 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:13.498903 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:15.181526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.181469 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn"] Oct 30 18:04:15.182070 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.182049 4603 cpu_manager.go:414] "containerMap: removing container" podUID="e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" containerName="istio-proxy" Oct 30 18:04:15.182158 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.182073 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2d00243-a6a6-438f-ab25-9ae2ac5e75bd" containerName="istio-proxy" Oct 30 18:04:15.182158 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.182085 4603 cpu_manager.go:414] "containerMap: removing container" podUID="fedfff5d-d5b3-4a6f-b126-34fb9c52ac61" containerName="maas-api" Oct 30 18:04:15.182158 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.182091 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fedfff5d-d5b3-4a6f-b126-34fb9c52ac61" containerName="maas-api" Oct 30 18:04:15.182315 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.182236 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="fedfff5d-d5b3-4a6f-b126-34fb9c52ac61" containerName="maas-api" Oct 30 18:04:15.185322 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.185302 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" Oct 30 18:04:15.199756 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.199736 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-3cc458-serving-cert\"" Oct 30 18:04:15.213640 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.213617 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn"] Oct 30 18:04:15.270318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.270287 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-maas-api-tls\") pod \"maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn\" (UID: \"e5c8b1f8-fba8-4afc-8887-8b1375d67b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" Oct 30 18:04:15.270497 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.270341 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j86mv\" (UniqueName: \"kubernetes.io/projected/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-kube-api-access-j86mv\") pod \"maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn\" (UID: \"e5c8b1f8-fba8-4afc-8887-8b1375d67b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" Oct 30 18:04:15.270497 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.270446 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn\" (UID: \"e5c8b1f8-fba8-4afc-8887-8b1375d67b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" Oct 30 18:04:15.270635 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.270561 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-metrics-certs\") pod \"maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn\" (UID: \"e5c8b1f8-fba8-4afc-8887-8b1375d67b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" Oct 30 18:04:15.314190 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.314154 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:15.371408 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.371377 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-metrics-certs\") pod \"maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn\" (UID: \"e5c8b1f8-fba8-4afc-8887-8b1375d67b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" Oct 30 18:04:15.371593 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.371458 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-maas-api-tls\") pod \"maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn\" (UID: \"e5c8b1f8-fba8-4afc-8887-8b1375d67b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" Oct 30 18:04:15.371593 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.371490 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j86mv\" (UniqueName: \"kubernetes.io/projected/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-kube-api-access-j86mv\") pod \"maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn\" (UID: \"e5c8b1f8-fba8-4afc-8887-8b1375d67b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" Oct 30 18:04:15.371593 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.371555 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn\" (UID: \"e5c8b1f8-fba8-4afc-8887-8b1375d67b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" Oct 30 18:04:15.372221 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.372198 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn\" (UID: \"e5c8b1f8-fba8-4afc-8887-8b1375d67b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" Oct 30 18:04:15.374119 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.374097 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-metrics-certs\") pod \"maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn\" (UID: \"e5c8b1f8-fba8-4afc-8887-8b1375d67b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" Oct 30 18:04:15.374119 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.374114 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-maas-api-tls\") pod \"maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn\" (UID: \"e5c8b1f8-fba8-4afc-8887-8b1375d67b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" Oct 30 18:04:15.394533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.394489 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j86mv\" (UniqueName: \"kubernetes.io/projected/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-kube-api-access-j86mv\") pod \"maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn\" (UID: \"e5c8b1f8-fba8-4afc-8887-8b1375d67b22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" Oct 30 18:04:15.463241 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.463122 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:15.496939 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.496907 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" Oct 30 18:04:15.648434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.648401 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:15.804303 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.804281 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn"] Oct 30 18:04:15.805958 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:04:15.805927 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c8b1f8_fba8_4afc_8887_8b1375d67b22.slice/crio-9543187e0b903fe1ffe1f366b05342c0f9e53eb43fb78a9f52c509d7c24e4f3c WatchSource:0}: Error finding container 9543187e0b903fe1ffe1f366b05342c0f9e53eb43fb78a9f52c509d7c24e4f3c: Status 404 returned error can't find the container with id 9543187e0b903fe1ffe1f366b05342c0f9e53eb43fb78a9f52c509d7c24e4f3c Oct 30 18:04:15.807341 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:15.807321 4603 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:04:16.805469 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:16.805432 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" event={"ID":"e5c8b1f8-fba8-4afc-8887-8b1375d67b22","Type":"ContainerStarted","Data":"764446c320c5550776b951f1283da60ce87998c86f8b69f0f4859a38b676db69"} Oct 30 18:04:16.805938 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:16.805485 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" event={"ID":"e5c8b1f8-fba8-4afc-8887-8b1375d67b22","Type":"ContainerStarted","Data":"9543187e0b903fe1ffe1f366b05342c0f9e53eb43fb78a9f52c509d7c24e4f3c"} Oct 30 18:04:16.805938 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:16.805622 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" Oct 30 18:04:16.849158 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:16.849108 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" podStartSLOduration=1.505328436 podStartE2EDuration="1.84909388s" totalImagesPullingTime="343.765444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:16.846782485 +0000 UTC m=+717.562352358" watchObservedRunningTime="2026-10-30 18:04:16.84909388 +0000 UTC m=+717.564663891" Oct 30 18:04:17.160522 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:17.160419 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:18.899039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:18.898990 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:20.133347 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:20.133311 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:04:20.653835 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:20.653800 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:21.938497 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:04:21.938451 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:04:22.345716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:22.345682 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk"] Oct 30 18:04:22.346224 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:22.346159 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" podUID="e355edcc-648d-492b-9665-2107f84afb63" containerName="istio-proxy" containerID="cri-o://d516d859312996808ec5ad34b36d545a649b85d0ac55ed9fc249e3f7221846d9" gracePeriod=30 Oct 30 18:04:22.417039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:22.417005 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:22.765667 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:22.765569 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:22.844099 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:22.844070 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" Oct 30 18:04:23.221558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.221518 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:23.631132 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.631101 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp"] Oct 30 18:04:23.638539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.638495 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp"] Oct 30 18:04:23.638776 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.638602 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.653441 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.653422 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-57ee15-openshift-default-dockercfg-jlxgq\"" Oct 30 18:04:23.682594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.682568 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.682702 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.682678 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.682743 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.682711 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.682743 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.682735 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.682842 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.682824 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.682877 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.682861 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.682911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.682880 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.682944 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.682906 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr8gp\" (UniqueName: \"kubernetes.io/projected/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-kube-api-access-xr8gp\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.683016 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.683002 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.688652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.688620 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:23.784528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.784087 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.784528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.784147 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.784528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.784177 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.784528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.784256 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.784528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.784284 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.784528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.784311 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.784528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.784354 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr8gp\" (UniqueName: \"kubernetes.io/projected/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-kube-api-access-xr8gp\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.784528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.784398 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.784528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.784440 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.785148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.784995 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.788576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.785252 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.788576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.785947 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.788576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.786209 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.788576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.786854 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.816341 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.816303 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.828414 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.828391 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.829958 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.829913 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.831217 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.831170 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr8gp\" (UniqueName: \"kubernetes.io/projected/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-kube-api-access-xr8gp\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:23.876373 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.876348 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:23.954145 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:23.954056 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:24.029529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:24.029485 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:24.316655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:24.310967 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp"] Oct 30 18:04:24.320694 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:04:24.320662 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744e9a72_c23c_4b8f_b0fa_ef7f791ce58a.slice/crio-79f4922a74814972e3fc9332fa7feb944ca14db75955fcaa9d793cbc4d8d435b WatchSource:0}: Error finding container 79f4922a74814972e3fc9332fa7feb944ca14db75955fcaa9d793cbc4d8d435b: Status 404 returned error can't find the container with id 79f4922a74814972e3fc9332fa7feb944ca14db75955fcaa9d793cbc4d8d435b Oct 30 18:04:24.323081 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:24.323052 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:04:24.323149 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:24.323119 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:04:24.323195 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:24.323162 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:04:24.810049 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:24.809963 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:24.862197 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:24.862156 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" event={"ID":"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a","Type":"ContainerStarted","Data":"eaa1fb3ae846f7041b3a2811b2d48bc6aaaeeaf45ef5e5873ac44693145d9a66"} Oct 30 18:04:24.862385 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:24.862227 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" event={"ID":"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a","Type":"ContainerStarted","Data":"79f4922a74814972e3fc9332fa7feb944ca14db75955fcaa9d793cbc4d8d435b"} Oct 30 18:04:24.954783 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:24.954754 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:25.444529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:25.442941 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:25.961142 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:25.961115 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:26.059271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:26.059217 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" podStartSLOduration=3.059204321 podStartE2EDuration="3.059204321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:24.912355958 +0000 UTC m=+725.627925833" watchObservedRunningTime="2026-10-30 18:04:26.059204321 +0000 UTC m=+726.774774195" Oct 30 18:04:26.228579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:26.228058 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:26.413531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:26.404030 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:26.872947 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:26.872924 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:26.873691 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:26.873672 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:26.934671 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:26.934636 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:27.210930 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.210848 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:27.514612 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.514521 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:27.644731 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.644705 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:04:27.725876 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.725793 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:27.784787 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.784752 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e355edcc-648d-492b-9665-2107f84afb63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e355edcc-648d-492b-9665-2107f84afb63-istiod-ca-cert\") pod \"e355edcc-648d-492b-9665-2107f84afb63\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " Oct 30 18:04:27.784981 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.784810 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-workload-socket\") pod \"e355edcc-648d-492b-9665-2107f84afb63\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " Oct 30 18:04:27.784981 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.784870 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-istio-data\") pod \"e355edcc-648d-492b-9665-2107f84afb63\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " Oct 30 18:04:27.784981 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.784893 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e355edcc-648d-492b-9665-2107f84afb63-kube-api-access-7lqkb\" (UniqueName: \"kubernetes.io/projected/e355edcc-648d-492b-9665-2107f84afb63-kube-api-access-7lqkb\") pod \"e355edcc-648d-492b-9665-2107f84afb63\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " Oct 30 18:04:27.784981 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.784960 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-istio-envoy\") pod \"e355edcc-648d-492b-9665-2107f84afb63\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " Oct 30 18:04:27.785198 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.784988 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e355edcc-648d-492b-9665-2107f84afb63-istio-token\" (UniqueName: \"kubernetes.io/projected/e355edcc-648d-492b-9665-2107f84afb63-istio-token\") pod \"e355edcc-648d-492b-9665-2107f84afb63\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " Oct 30 18:04:27.785198 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.785050 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-credential-socket\") pod \"e355edcc-648d-492b-9665-2107f84afb63\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " Oct 30 18:04:27.785198 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.785111 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e355edcc-648d-492b-9665-2107f84afb63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e355edcc-648d-492b-9665-2107f84afb63-istio-podinfo\") pod \"e355edcc-648d-492b-9665-2107f84afb63\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " Oct 30 18:04:27.785198 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.785142 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-workload-certs\") pod \"e355edcc-648d-492b-9665-2107f84afb63\" (UID: \"e355edcc-648d-492b-9665-2107f84afb63\") " Oct 30 18:04:27.785412 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.785204 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e355edcc-648d-492b-9665-2107f84afb63-istiod-ca-cert" pod "e355edcc-648d-492b-9665-2107f84afb63" (UID: "e355edcc-648d-492b-9665-2107f84afb63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:04:27.785412 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.785213 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-istio-data" pod "e355edcc-648d-492b-9665-2107f84afb63" (UID: "e355edcc-648d-492b-9665-2107f84afb63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:27.785412 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.785228 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-workload-socket" pod "e355edcc-648d-492b-9665-2107f84afb63" (UID: "e355edcc-648d-492b-9665-2107f84afb63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:27.785576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.785461 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-credential-socket" pod "e355edcc-648d-492b-9665-2107f84afb63" (UID: "e355edcc-648d-492b-9665-2107f84afb63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:27.785576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.785539 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:04:27.785576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.785560 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e355edcc-648d-492b-9665-2107f84afb63-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:04:27.785576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.785573 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:04:27.785925 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.785894 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-workload-certs" pod "e355edcc-648d-492b-9665-2107f84afb63" (UID: "e355edcc-648d-492b-9665-2107f84afb63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:27.787626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.787601 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e355edcc-648d-492b-9665-2107f84afb63-kube-api-access-7lqkb" pod "e355edcc-648d-492b-9665-2107f84afb63" (UID: "e355edcc-648d-492b-9665-2107f84afb63"). InnerVolumeSpecName "kube-api-access-7lqkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:27.787903 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.787878 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-istio-envoy" pod "e355edcc-648d-492b-9665-2107f84afb63" (UID: "e355edcc-648d-492b-9665-2107f84afb63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:27.788099 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.788079 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e355edcc-648d-492b-9665-2107f84afb63-istio-token" pod "e355edcc-648d-492b-9665-2107f84afb63" (UID: "e355edcc-648d-492b-9665-2107f84afb63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:27.788285 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.788269 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e355edcc-648d-492b-9665-2107f84afb63-istio-podinfo" pod "e355edcc-648d-492b-9665-2107f84afb63" (UID: "e355edcc-648d-492b-9665-2107f84afb63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:04:27.879449 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.879411 4603 generic.go:356] "Generic (PLEG): container finished" podID="e355edcc-648d-492b-9665-2107f84afb63" containerID="d516d859312996808ec5ad34b36d545a649b85d0ac55ed9fc249e3f7221846d9" exitCode=0 Oct 30 18:04:27.879917 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.879468 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" Oct 30 18:04:27.879917 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.879531 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" event={"ID":"e355edcc-648d-492b-9665-2107f84afb63","Type":"ContainerDied","Data":"d516d859312996808ec5ad34b36d545a649b85d0ac55ed9fc249e3f7221846d9"} Oct 30 18:04:27.879917 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.879606 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk" event={"ID":"e355edcc-648d-492b-9665-2107f84afb63","Type":"ContainerDied","Data":"9a5f6f1967b58bfe1df2a82e45e0bad140ecceef73f0bb53d66b45bcb66dc93a"} Oct 30 18:04:27.879917 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.879636 4603 scope.go:122] "RemoveContainer" containerID="d516d859312996808ec5ad34b36d545a649b85d0ac55ed9fc249e3f7221846d9" Oct 30 18:04:27.886494 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.886465 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:04:27.886648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.886514 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e355edcc-648d-492b-9665-2107f84afb63-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:04:27.886648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.886530 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:04:27.886648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.886544 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lqkb\" (UniqueName: \"kubernetes.io/projected/e355edcc-648d-492b-9665-2107f84afb63-kube-api-access-7lqkb\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:04:27.886648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.886558 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e355edcc-648d-492b-9665-2107f84afb63-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:04:27.886648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.886572 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e355edcc-648d-492b-9665-2107f84afb63-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:04:27.914668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.912339 4603 scope.go:122] "RemoveContainer" containerID="2258b6a8808bef06a850862f288d2b4769736952d4d1efe9f87a65145af06b69" Oct 30 18:04:27.924213 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.924196 4603 scope.go:122] "RemoveContainer" containerID="d516d859312996808ec5ad34b36d545a649b85d0ac55ed9fc249e3f7221846d9" Oct 30 18:04:27.924490 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:04:27.924465 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d516d859312996808ec5ad34b36d545a649b85d0ac55ed9fc249e3f7221846d9\": container with ID starting with d516d859312996808ec5ad34b36d545a649b85d0ac55ed9fc249e3f7221846d9 not found: ID does not exist" containerID="d516d859312996808ec5ad34b36d545a649b85d0ac55ed9fc249e3f7221846d9" Oct 30 18:04:27.924629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.924515 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d516d859312996808ec5ad34b36d545a649b85d0ac55ed9fc249e3f7221846d9"} err="failed to get container status \"d516d859312996808ec5ad34b36d545a649b85d0ac55ed9fc249e3f7221846d9\": rpc error: code = NotFound desc = could not find container \"d516d859312996808ec5ad34b36d545a649b85d0ac55ed9fc249e3f7221846d9\": container with ID starting with d516d859312996808ec5ad34b36d545a649b85d0ac55ed9fc249e3f7221846d9 not found: ID does not exist" Oct 30 18:04:27.924629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.924539 4603 scope.go:122] "RemoveContainer" containerID="2258b6a8808bef06a850862f288d2b4769736952d4d1efe9f87a65145af06b69" Oct 30 18:04:27.924886 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:04:27.924863 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2258b6a8808bef06a850862f288d2b4769736952d4d1efe9f87a65145af06b69\": container with ID starting with 2258b6a8808bef06a850862f288d2b4769736952d4d1efe9f87a65145af06b69 not found: ID does not exist" containerID="2258b6a8808bef06a850862f288d2b4769736952d4d1efe9f87a65145af06b69" Oct 30 18:04:27.924958 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.924897 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2258b6a8808bef06a850862f288d2b4769736952d4d1efe9f87a65145af06b69"} err="failed to get container status \"2258b6a8808bef06a850862f288d2b4769736952d4d1efe9f87a65145af06b69\": rpc error: code = NotFound desc = could not find container \"2258b6a8808bef06a850862f288d2b4769736952d4d1efe9f87a65145af06b69\": container with ID starting with 2258b6a8808bef06a850862f288d2b4769736952d4d1efe9f87a65145af06b69 not found: ID does not exist" Oct 30 18:04:27.955359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.955328 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk"] Oct 30 18:04:27.964633 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:27.964606 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a9a40825-openshift-default-c6d9b777c-qr8gk"] Oct 30 18:04:29.117189 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.117130 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:29.386838 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.386753 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:29.424199 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.424171 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e355edcc-648d-492b-9665-2107f84afb63" path="/var/lib/kubelet/pods/e355edcc-648d-492b-9665-2107f84afb63/volumes" Oct 30 18:04:29.480127 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.480101 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp"] Oct 30 18:04:29.480523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.480478 4603 cpu_manager.go:414] "containerMap: removing container" podUID="e355edcc-648d-492b-9665-2107f84afb63" containerName="istio-proxy" Oct 30 18:04:29.480523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.480496 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e355edcc-648d-492b-9665-2107f84afb63" containerName="istio-proxy" Oct 30 18:04:29.480882 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.480650 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="e355edcc-648d-492b-9665-2107f84afb63" containerName="istio-proxy" Oct 30 18:04:29.480882 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.480675 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="e355edcc-648d-492b-9665-2107f84afb63" containerName="istio-proxy" Oct 30 18:04:29.484323 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.484302 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp" Oct 30 18:04:29.515544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.514970 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-57ee15\"" Oct 30 18:04:29.526917 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.526894 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-57ee15-dockercfg-m4bpf\"" Oct 30 18:04:29.528758 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.528735 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp"] Oct 30 18:04:29.540278 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.540256 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt"] Oct 30 18:04:29.540637 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.540624 4603 cpu_manager.go:414] "containerMap: removing container" podUID="e355edcc-648d-492b-9665-2107f84afb63" containerName="istio-proxy" Oct 30 18:04:29.540709 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.540638 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e355edcc-648d-492b-9665-2107f84afb63" containerName="istio-proxy" Oct 30 18:04:29.542773 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.542759 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt" Oct 30 18:04:29.570813 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.570786 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt"] Oct 30 18:04:29.571398 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.571368 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:29.645591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.641273 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7be351b-62cc-4555-84fa-a0f018a99059-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt\" (UID: \"c7be351b-62cc-4555-84fa-a0f018a99059\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt" Oct 30 18:04:29.645591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.641352 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f308b20b-d5d6-41db-8376-9cb9413b13cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp\" (UID: \"f308b20b-d5d6-41db-8376-9cb9413b13cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp" Oct 30 18:04:29.645591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.641398 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg9fm\" (UniqueName: \"kubernetes.io/projected/c7be351b-62cc-4555-84fa-a0f018a99059-kube-api-access-zg9fm\") pod \"payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt\" (UID: \"c7be351b-62cc-4555-84fa-a0f018a99059\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt" Oct 30 18:04:29.645591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.641434 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmwtq\" (UniqueName: \"kubernetes.io/projected/f308b20b-d5d6-41db-8376-9cb9413b13cf-kube-api-access-jmwtq\") pod \"payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp\" (UID: \"f308b20b-d5d6-41db-8376-9cb9413b13cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp" Oct 30 18:04:29.742141 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.742099 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7be351b-62cc-4555-84fa-a0f018a99059-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt\" (UID: \"c7be351b-62cc-4555-84fa-a0f018a99059\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt" Oct 30 18:04:29.742141 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.742151 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f308b20b-d5d6-41db-8376-9cb9413b13cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp\" (UID: \"f308b20b-d5d6-41db-8376-9cb9413b13cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp" Oct 30 18:04:29.742415 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.742181 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg9fm\" (UniqueName: \"kubernetes.io/projected/c7be351b-62cc-4555-84fa-a0f018a99059-kube-api-access-zg9fm\") pod \"payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt\" (UID: \"c7be351b-62cc-4555-84fa-a0f018a99059\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt" Oct 30 18:04:29.742415 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.742204 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmwtq\" (UniqueName: \"kubernetes.io/projected/f308b20b-d5d6-41db-8376-9cb9413b13cf-kube-api-access-jmwtq\") pod \"payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp\" (UID: \"f308b20b-d5d6-41db-8376-9cb9413b13cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp" Oct 30 18:04:29.743113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.743087 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f308b20b-d5d6-41db-8376-9cb9413b13cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp\" (UID: \"f308b20b-d5d6-41db-8376-9cb9413b13cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp" Oct 30 18:04:29.743363 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.743343 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7be351b-62cc-4555-84fa-a0f018a99059-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt\" (UID: \"c7be351b-62cc-4555-84fa-a0f018a99059\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt" Oct 30 18:04:29.773822 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.773794 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmwtq\" (UniqueName: \"kubernetes.io/projected/f308b20b-d5d6-41db-8376-9cb9413b13cf-kube-api-access-jmwtq\") pod \"payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp\" (UID: \"f308b20b-d5d6-41db-8376-9cb9413b13cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp" Oct 30 18:04:29.773921 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.773874 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg9fm\" (UniqueName: \"kubernetes.io/projected/c7be351b-62cc-4555-84fa-a0f018a99059-kube-api-access-zg9fm\") pod \"payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt\" (UID: \"c7be351b-62cc-4555-84fa-a0f018a99059\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt" Oct 30 18:04:29.826167 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.826134 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp" Oct 30 18:04:29.854214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:29.854182 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt" Oct 30 18:04:30.134104 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:30.134082 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp"] Oct 30 18:04:30.136103 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:04:30.136074 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf308b20b_d5d6_41db_8376_9cb9413b13cf.slice/crio-731b04f8d69d055f825f9887a17c09397e23236db2d77c3a8427e43d67f048a7 WatchSource:0}: Error finding container 731b04f8d69d055f825f9887a17c09397e23236db2d77c3a8427e43d67f048a7: Status 404 returned error can't find the container with id 731b04f8d69d055f825f9887a17c09397e23236db2d77c3a8427e43d67f048a7 Oct 30 18:04:30.214876 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:30.212954 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt"] Oct 30 18:04:30.218440 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:04:30.218414 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7be351b_62cc_4555_84fa_a0f018a99059.slice/crio-ef53ab62af9d7e8fd12a9fa0e49d6b863d10195fd32a13bcb132105a2394d975 WatchSource:0}: Error finding container ef53ab62af9d7e8fd12a9fa0e49d6b863d10195fd32a13bcb132105a2394d975: Status 404 returned error can't find the container with id ef53ab62af9d7e8fd12a9fa0e49d6b863d10195fd32a13bcb132105a2394d975 Oct 30 18:04:30.932685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:30.932612 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp" event={"ID":"f308b20b-d5d6-41db-8376-9cb9413b13cf","Type":"ContainerStarted","Data":"731b04f8d69d055f825f9887a17c09397e23236db2d77c3a8427e43d67f048a7"} Oct 30 18:04:30.937861 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:30.937820 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt" event={"ID":"c7be351b-62cc-4555-84fa-a0f018a99059","Type":"ContainerStarted","Data":"ef53ab62af9d7e8fd12a9fa0e49d6b863d10195fd32a13bcb132105a2394d975"} Oct 30 18:04:30.981545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:30.981483 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:31.745347 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:31.745317 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:04:31.974574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:31.973729 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:31.995945 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:31.995864 4603 generic.go:356] "Generic (PLEG): container finished" podID="e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" containerID="53a9ec2a86a85da94835ae4ea315d8040fd15baad1f9e73f353dd19c01b217a3" exitCode=0 Oct 30 18:04:31.996143 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:31.996010 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" event={"ID":"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685","Type":"ContainerDied","Data":"53a9ec2a86a85da94835ae4ea315d8040fd15baad1f9e73f353dd19c01b217a3"} Oct 30 18:04:31.996490 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:31.996470 4603 scope.go:122] "RemoveContainer" containerID="53a9ec2a86a85da94835ae4ea315d8040fd15baad1f9e73f353dd19c01b217a3" Oct 30 18:04:31.998895 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:31.998662 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:04:31.998895 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:31.998755 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:04:31.998895 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:31.998808 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:04:32.623627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:32.623596 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:32.850563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:32.850099 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:33.058533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:33.057632 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" event={"ID":"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685","Type":"ContainerStarted","Data":"6d9ee8906105d776e54d58e53db8f078fd097dc052ea82e85fb374224a3319ea"} Oct 30 18:04:33.079723 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:33.079633 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:04:33.081809 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:33.080411 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 18:04:33.081809 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:33.080521 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" podUID="e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 18:04:34.064574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:34.064536 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp" event={"ID":"f308b20b-d5d6-41db-8376-9cb9413b13cf","Type":"ContainerStarted","Data":"82015ea5d91c95a00fd537a325bb2e8bd201362ee53b0e30fbce799555e309eb"} Oct 30 18:04:34.065057 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:34.064650 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp" Oct 30 18:04:34.066276 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:34.066256 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt" event={"ID":"c7be351b-62cc-4555-84fa-a0f018a99059","Type":"ContainerStarted","Data":"3a635d0121df94235e9d90badb1b899ad0a782b669c9908bf48aa8840d77c213"} Oct 30 18:04:34.066480 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:34.066463 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt" Oct 30 18:04:34.071865 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:34.071848 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 18:04:34.071955 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:34.071886 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" podUID="e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 18:04:34.155187 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:34.155137 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp" podStartSLOduration=1.611897087 podStartE2EDuration="5.155123788s" totalImagesPullingTime="3.543226701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:34.115182461 +0000 UTC m=+734.830752335" watchObservedRunningTime="2026-10-30 18:04:34.155123788 +0000 UTC m=+734.870693699" Oct 30 18:04:34.155777 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:34.155752 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt" podStartSLOduration=1.820129778 podStartE2EDuration="5.15574209s" totalImagesPullingTime="3.335612312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:34.154279857 +0000 UTC m=+734.869849731" watchObservedRunningTime="2026-10-30 18:04:34.15574209 +0000 UTC m=+734.871312005" Oct 30 18:04:34.566553 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:34.566518 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:35.071530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:35.071485 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 18:04:35.071980 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:35.071547 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" podUID="e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 18:04:35.571596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:35.571574 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:04:36.071748 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.071713 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 18:04:36.072148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.071781 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" podUID="e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 18:04:36.077323 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.077296 4603 generic.go:356] "Generic (PLEG): container finished" podID="22e0b280-2b65-4fad-8344-67049190d108" containerID="16d8aa22cecbcf7f4440eecd90d2bc6e688028c1e367f0d43f4f2c01b4726a68" exitCode=0 Oct 30 18:04:36.077454 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.077368 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" event={"ID":"22e0b280-2b65-4fad-8344-67049190d108","Type":"ContainerDied","Data":"16d8aa22cecbcf7f4440eecd90d2bc6e688028c1e367f0d43f4f2c01b4726a68"} Oct 30 18:04:36.077735 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.077718 4603 scope.go:122] "RemoveContainer" containerID="16d8aa22cecbcf7f4440eecd90d2bc6e688028c1e367f0d43f4f2c01b4726a68" Oct 30 18:04:36.078999 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.078973 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:04:36.079059 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.079037 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:04:36.079098 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.079071 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:04:36.126960 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.126933 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2"] Oct 30 18:04:36.130311 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.130282 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" Oct 30 18:04:36.146694 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.146668 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-57ee15-serving-cert\"" Oct 30 18:04:36.148552 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.148529 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2"] Oct 30 18:04:36.163738 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.163710 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7275\" (UniqueName: \"kubernetes.io/projected/e0c05c4b-da86-4622-88ec-960248babce5-kube-api-access-h7275\") pod \"maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2\" (UID: \"e0c05c4b-da86-4622-88ec-960248babce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" Oct 30 18:04:36.165127 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.165100 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c05c4b-da86-4622-88ec-960248babce5-maas-api-tls\") pod \"maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2\" (UID: \"e0c05c4b-da86-4622-88ec-960248babce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" Oct 30 18:04:36.165374 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.165354 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0c05c4b-da86-4622-88ec-960248babce5-metrics-certs\") pod \"maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2\" (UID: \"e0c05c4b-da86-4622-88ec-960248babce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" Oct 30 18:04:36.165749 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.165706 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c05c4b-da86-4622-88ec-960248babce5-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2\" (UID: \"e0c05c4b-da86-4622-88ec-960248babce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" Oct 30 18:04:36.241092 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.241053 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:36.267043 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.267007 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c05c4b-da86-4622-88ec-960248babce5-maas-api-tls\") pod \"maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2\" (UID: \"e0c05c4b-da86-4622-88ec-960248babce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" Oct 30 18:04:36.267186 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.267099 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0c05c4b-da86-4622-88ec-960248babce5-metrics-certs\") pod \"maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2\" (UID: \"e0c05c4b-da86-4622-88ec-960248babce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" Oct 30 18:04:36.267186 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.267130 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c05c4b-da86-4622-88ec-960248babce5-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2\" (UID: \"e0c05c4b-da86-4622-88ec-960248babce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" Oct 30 18:04:36.267257 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.267201 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7275\" (UniqueName: \"kubernetes.io/projected/e0c05c4b-da86-4622-88ec-960248babce5-kube-api-access-h7275\") pod \"maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2\" (UID: \"e0c05c4b-da86-4622-88ec-960248babce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" Oct 30 18:04:36.268258 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.268226 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c05c4b-da86-4622-88ec-960248babce5-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2\" (UID: \"e0c05c4b-da86-4622-88ec-960248babce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" Oct 30 18:04:36.269540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.269520 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c05c4b-da86-4622-88ec-960248babce5-maas-api-tls\") pod \"maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2\" (UID: \"e0c05c4b-da86-4622-88ec-960248babce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" Oct 30 18:04:36.270010 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.269993 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0c05c4b-da86-4622-88ec-960248babce5-metrics-certs\") pod \"maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2\" (UID: \"e0c05c4b-da86-4622-88ec-960248babce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" Oct 30 18:04:36.314393 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.314364 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7275\" (UniqueName: \"kubernetes.io/projected/e0c05c4b-da86-4622-88ec-960248babce5-kube-api-access-h7275\") pod \"maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2\" (UID: \"e0c05c4b-da86-4622-88ec-960248babce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" Oct 30 18:04:36.409036 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.408924 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:36.445365 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.445338 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" Oct 30 18:04:36.779750 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.779204 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2"] Oct 30 18:04:36.783541 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:04:36.781720 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c05c4b_da86_4622_88ec_960248babce5.slice/crio-c88b02632211660a408b58a25e98da5733ed5f0ffa391b00348dca39ef48895d WatchSource:0}: Error finding container c88b02632211660a408b58a25e98da5733ed5f0ffa391b00348dca39ef48895d: Status 404 returned error can't find the container with id c88b02632211660a408b58a25e98da5733ed5f0ffa391b00348dca39ef48895d Oct 30 18:04:36.987515 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:36.987473 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:37.078575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:37.077038 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 18:04:37.078575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:37.077142 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" podUID="e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 18:04:37.098526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:37.098345 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" event={"ID":"e0c05c4b-da86-4622-88ec-960248babce5","Type":"ContainerStarted","Data":"c88b02632211660a408b58a25e98da5733ed5f0ffa391b00348dca39ef48895d"} Oct 30 18:04:37.119530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:37.106675 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" event={"ID":"22e0b280-2b65-4fad-8344-67049190d108","Type":"ContainerStarted","Data":"987b8014d0d4886b3ebdf1c880a7bb694d1169d0fd45406d4d558b7cc98eda45"} Oct 30 18:04:37.156538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:37.148476 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:37.342797 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:37.341922 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:38.041664 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:38.041592 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:04:38.041911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:38.041884 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:04:38.042052 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:38.041929 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" podUID="22e0b280-2b65-4fad-8344-67049190d108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 18:04:38.073091 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:38.073061 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 18:04:38.073212 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:38.073112 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" podUID="e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 18:04:38.136080 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:38.135835 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" event={"ID":"e0c05c4b-da86-4622-88ec-960248babce5","Type":"ContainerStarted","Data":"0124c9526b3d7458ed3417d383b5321f48f66eb01120b4bfc1ed62f6e1cc5f04"} Oct 30 18:04:38.136080 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:38.135926 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" Oct 30 18:04:38.154816 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:38.154742 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:39.036226 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:39.036195 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:04:39.036399 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:39.036250 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" podUID="22e0b280-2b65-4fad-8344-67049190d108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 18:04:39.075562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:39.075533 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:04:39.076469 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:39.076448 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:04:39.156311 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:39.156265 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" podStartSLOduration=2.761049039 podStartE2EDuration="3.15625043s" totalImagesPullingTime="395.201391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:38.199377588 +0000 UTC m=+738.914947463" watchObservedRunningTime="2026-10-30 18:04:39.15625043 +0000 UTC m=+739.871820304" Oct 30 18:04:39.330715 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:39.330603 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:39.859910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:39.859872 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:04:39.972744 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:39.972715 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:40.035944 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:40.035908 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:04:40.036273 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:40.036243 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" podUID="22e0b280-2b65-4fad-8344-67049190d108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 18:04:40.623851 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:40.623817 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:04:41.038007 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:41.037971 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:04:41.038304 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:41.038274 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" podUID="22e0b280-2b65-4fad-8344-67049190d108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 18:04:41.656529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:41.656234 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:42.036527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:42.035919 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:04:42.036527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:42.035981 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" podUID="22e0b280-2b65-4fad-8344-67049190d108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 18:04:43.039517 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:43.039473 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:04:43.040668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:43.040648 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:04:43.488211 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:43.488124 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:44.143798 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:44.143771 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" Oct 30 18:04:45.074615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:45.074588 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt" Oct 30 18:04:45.074815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:45.074654 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp" Oct 30 18:04:45.373309 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:45.373214 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:46.016141 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.016106 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv"] Oct 30 18:04:46.020986 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.020959 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:46.037531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.037477 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b424a5\"/\"default-dockercfg-4tgq7\"" Oct 30 18:04:46.039120 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.039096 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-dshm\") pod \"simulator-w1-b424a5-kserve-7d56bc98c8-56hfv\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:46.039368 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.039349 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-tls-certs\") pod \"simulator-w1-b424a5-kserve-7d56bc98c8-56hfv\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:46.039603 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.039586 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-model-cache\") pod \"simulator-w1-b424a5-kserve-7d56bc98c8-56hfv\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:46.039839 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.039816 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-tmp-dir\") pod \"simulator-w1-b424a5-kserve-7d56bc98c8-56hfv\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:46.039945 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.039851 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkk9w\" (UniqueName: \"kubernetes.io/projected/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-kube-api-access-dkk9w\") pod \"simulator-w1-b424a5-kserve-7d56bc98c8-56hfv\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:46.040018 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.039974 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-home\") pod \"simulator-w1-b424a5-kserve-7d56bc98c8-56hfv\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:46.049252 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.049221 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b424a5\"/\"simulator-w1-b424a5-kserve-self-signed-certs\"" Oct 30 18:04:46.049369 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.049357 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b424a5\"/\"openshift-service-ca.crt\"" Oct 30 18:04:46.049458 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.049443 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b424a5\"/\"kube-root-ca.crt\"" Oct 30 18:04:46.051577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.051556 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv"] Oct 30 18:04:46.121121 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.121086 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:46.141255 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.141218 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-dshm\") pod \"simulator-w1-b424a5-kserve-7d56bc98c8-56hfv\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:46.141255 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.141253 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-tls-certs\") pod \"simulator-w1-b424a5-kserve-7d56bc98c8-56hfv\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:46.141498 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.141288 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-model-cache\") pod \"simulator-w1-b424a5-kserve-7d56bc98c8-56hfv\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:46.141498 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.141463 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-tmp-dir\") pod \"simulator-w1-b424a5-kserve-7d56bc98c8-56hfv\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:46.141632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.141521 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkk9w\" (UniqueName: \"kubernetes.io/projected/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-kube-api-access-dkk9w\") pod \"simulator-w1-b424a5-kserve-7d56bc98c8-56hfv\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:46.141632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.141619 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-home\") pod \"simulator-w1-b424a5-kserve-7d56bc98c8-56hfv\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:46.141767 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.141747 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-model-cache\") pod \"simulator-w1-b424a5-kserve-7d56bc98c8-56hfv\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:46.141838 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.141816 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-tmp-dir\") pod \"simulator-w1-b424a5-kserve-7d56bc98c8-56hfv\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:46.142055 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.142009 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-home\") pod \"simulator-w1-b424a5-kserve-7d56bc98c8-56hfv\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:46.143679 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.143656 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-dshm\") pod \"simulator-w1-b424a5-kserve-7d56bc98c8-56hfv\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:46.143973 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.143958 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-tls-certs\") pod \"simulator-w1-b424a5-kserve-7d56bc98c8-56hfv\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:46.165401 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.165368 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkk9w\" (UniqueName: \"kubernetes.io/projected/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-kube-api-access-dkk9w\") pod \"simulator-w1-b424a5-kserve-7d56bc98c8-56hfv\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:46.334418 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.334385 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:46.635836 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:46.635813 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv"] Oct 30 18:04:46.638014 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:04:46.637991 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea5f82db_b4a7_4c43_8131_dcffe2baa65b.slice/crio-e24ffe5d7ffda84cbfc3145069cfb606f164d0f532fe1e7f8ed35bf9eed205b6 WatchSource:0}: Error finding container e24ffe5d7ffda84cbfc3145069cfb606f164d0f532fe1e7f8ed35bf9eed205b6: Status 404 returned error can't find the container with id e24ffe5d7ffda84cbfc3145069cfb606f164d0f532fe1e7f8ed35bf9eed205b6 Oct 30 18:04:47.175048 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:47.172742 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:47.233148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:47.233103 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" event={"ID":"ea5f82db-b4a7-4c43-8131-dcffe2baa65b","Type":"ContainerStarted","Data":"f0f518fa4ff68de6a8fd17fc6c8e60bff8745c593d4c0669319e72ec6fdaeeaa"} Oct 30 18:04:47.233365 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:47.233162 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" event={"ID":"ea5f82db-b4a7-4c43-8131-dcffe2baa65b","Type":"ContainerStarted","Data":"e24ffe5d7ffda84cbfc3145069cfb606f164d0f532fe1e7f8ed35bf9eed205b6"} Oct 30 18:04:47.276109 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:47.276053 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podStartSLOduration=2.276034265 podStartE2EDuration="2.276034265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:47.274647595 +0000 UTC m=+747.990217482" watchObservedRunningTime="2026-10-30 18:04:47.276034265 +0000 UTC m=+747.991604139" Oct 30 18:04:47.334926 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:47.334889 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:49.076388 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:49.076347 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:49.572105 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:49.572077 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:04:50.250523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:50.250467 4603 generic.go:356] "Generic (PLEG): container finished" podID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerID="eaa1fb3ae846f7041b3a2811b2d48bc6aaaeeaf45ef5e5873ac44693145d9a66" exitCode=0 Oct 30 18:04:50.251218 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:50.251139 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" event={"ID":"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a","Type":"ContainerDied","Data":"eaa1fb3ae846f7041b3a2811b2d48bc6aaaeeaf45ef5e5873ac44693145d9a66"} Oct 30 18:04:50.252116 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:50.252062 4603 scope.go:122] "RemoveContainer" containerID="eaa1fb3ae846f7041b3a2811b2d48bc6aaaeeaf45ef5e5873ac44693145d9a66" Oct 30 18:04:50.254640 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:50.254555 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:04:50.254922 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:50.254886 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:04:50.255098 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:50.255079 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:04:51.267674 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:51.267635 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" event={"ID":"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a","Type":"ContainerStarted","Data":"512a4781daf6c9dda86706fb8577f67604b2aa0887f8a061d5ba0f82bf8be56f"} Oct 30 18:04:51.370707 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:51.370659 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:51.938275 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:04:51.938241 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:04:51.954549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:51.954530 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:51.954806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:51.954782 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 18:04:51.954908 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:51.954830 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" podUID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 18:04:52.655690 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:52.655657 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:52.954481 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:52.954373 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 18:04:52.954481 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:52.954442 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" podUID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 18:04:53.040809 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:53.040776 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:53.247990 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:53.247900 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:53.430956 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:53.430870 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:53.954864 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:53.954833 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 18:04:53.955275 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:53.954887 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" podUID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 18:04:54.347031 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:54.346991 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:54.806733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:54.806700 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:54.955207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:54.955175 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 18:04:54.955635 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:54.955232 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" podUID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 18:04:55.956985 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:55.956949 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 18:04:55.957732 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:55.957700 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" podUID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 18:04:56.334587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:56.334544 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:56.347080 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:56.347052 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:56.667117 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:56.667031 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:56.873285 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:56.873260 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:56.959439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:56.959380 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:57.284915 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.284883 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq"] Oct 30 18:04:57.288253 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.288224 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:04:57.301635 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.301614 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:04:57.309019 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.309002 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-419d32\"/\"kube-root-ca.crt\"" Oct 30 18:04:57.309280 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.309264 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-419d32\"/\"openshift-service-ca.crt\"" Oct 30 18:04:57.313397 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.313378 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:04:57.317183 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.317165 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-419d32\"/\"default-dockercfg-5z8s9\"" Oct 30 18:04:57.317286 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.317257 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-419d32\"/\"premium-w4-419d32-kserve-self-signed-certs\"" Oct 30 18:04:57.319245 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.319227 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq"] Oct 30 18:04:57.394605 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.394573 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-dshm\") pod \"premium-w4-419d32-kserve-6b55bf5ccf-q7ddq\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:04:57.394771 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.394641 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-model-cache\") pod \"premium-w4-419d32-kserve-6b55bf5ccf-q7ddq\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:04:57.394771 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.394751 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcss4\" (UniqueName: \"kubernetes.io/projected/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-kube-api-access-zcss4\") pod \"premium-w4-419d32-kserve-6b55bf5ccf-q7ddq\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:04:57.394879 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.394799 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-tls-certs\") pod \"premium-w4-419d32-kserve-6b55bf5ccf-q7ddq\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:04:57.395026 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.395007 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-tmp-dir\") pod \"premium-w4-419d32-kserve-6b55bf5ccf-q7ddq\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:04:57.395129 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.395110 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-home\") pod \"premium-w4-419d32-kserve-6b55bf5ccf-q7ddq\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:04:57.411224 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.411197 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:57.495792 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.495761 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-model-cache\") pod \"premium-w4-419d32-kserve-6b55bf5ccf-q7ddq\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:04:57.495792 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.495796 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcss4\" (UniqueName: \"kubernetes.io/projected/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-kube-api-access-zcss4\") pod \"premium-w4-419d32-kserve-6b55bf5ccf-q7ddq\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:04:57.496033 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.495820 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-tls-certs\") pod \"premium-w4-419d32-kserve-6b55bf5ccf-q7ddq\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:04:57.496033 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.495890 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-tmp-dir\") pod \"premium-w4-419d32-kserve-6b55bf5ccf-q7ddq\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:04:57.496033 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.495935 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-home\") pod \"premium-w4-419d32-kserve-6b55bf5ccf-q7ddq\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:04:57.496033 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.495976 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-dshm\") pod \"premium-w4-419d32-kserve-6b55bf5ccf-q7ddq\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:04:57.496265 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.496132 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-model-cache\") pod \"premium-w4-419d32-kserve-6b55bf5ccf-q7ddq\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:04:57.496316 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.496279 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-tmp-dir\") pod \"premium-w4-419d32-kserve-6b55bf5ccf-q7ddq\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:04:57.496316 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.496310 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-home\") pod \"premium-w4-419d32-kserve-6b55bf5ccf-q7ddq\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:04:57.498257 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.498234 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-dshm\") pod \"premium-w4-419d32-kserve-6b55bf5ccf-q7ddq\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:04:57.498354 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.498336 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-tls-certs\") pod \"premium-w4-419d32-kserve-6b55bf5ccf-q7ddq\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:04:57.518366 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.518340 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcss4\" (UniqueName: \"kubernetes.io/projected/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-kube-api-access-zcss4\") pod \"premium-w4-419d32-kserve-6b55bf5ccf-q7ddq\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:04:57.603017 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.602695 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:04:57.912992 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:57.912972 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq"] Oct 30 18:04:57.915019 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:04:57.914934 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e0d5196_98c7_4b7c_b045_623b42c2e3d0.slice/crio-6043ca5f32e0efbc365ed7aec4e6e58d9630a261ee14d06afa58a3279caf4870 WatchSource:0}: Error finding container 6043ca5f32e0efbc365ed7aec4e6e58d9630a261ee14d06afa58a3279caf4870: Status 404 returned error can't find the container with id 6043ca5f32e0efbc365ed7aec4e6e58d9630a261ee14d06afa58a3279caf4870 Oct 30 18:04:58.307103 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:58.307071 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" event={"ID":"1e0d5196-98c7-4b7c-b045-623b42c2e3d0","Type":"ContainerStarted","Data":"a37f1ac4eaa4cef7dea486ac59ac985bacbd9661f05e750063ce8036c5b151d9"} Oct 30 18:04:58.307487 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:58.307117 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" event={"ID":"1e0d5196-98c7-4b7c-b045-623b42c2e3d0","Type":"ContainerStarted","Data":"6043ca5f32e0efbc365ed7aec4e6e58d9630a261ee14d06afa58a3279caf4870"} Oct 30 18:04:58.348832 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:58.348783 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podStartSLOduration=1.348770196 podStartE2EDuration="1.348770196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:58.347206234 +0000 UTC m=+759.062776105" watchObservedRunningTime="2026-10-30 18:04:58.348770196 +0000 UTC m=+759.064340070" Oct 30 18:04:58.456413 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:58.456385 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:58.604707 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:04:58.604634 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:05:00.055520 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:00.055472 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:01.164081 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.164050 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm"] Oct 30 18:05:01.166705 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.166680 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.188820 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.188801 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-419d32-openshift-default-dockercfg-l4pzs\"" Oct 30 18:05:01.209839 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.209815 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm"] Oct 30 18:05:01.222447 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.222414 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:05:01.245095 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.245075 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.245206 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.245102 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.245206 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.245194 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.245326 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.245289 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.245379 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.245365 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.245446 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.245426 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.245524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.245474 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lfq6\" (UniqueName: \"kubernetes.io/projected/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-kube-api-access-5lfq6\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.245589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.245541 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.245589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.245571 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.346082 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.346052 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.346224 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.346092 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.346224 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.346119 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lfq6\" (UniqueName: \"kubernetes.io/projected/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-kube-api-access-5lfq6\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.346224 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.346164 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.346224 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.346190 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.346438 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.346271 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.346438 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.346299 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.346438 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.346387 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.346638 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.346454 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.346638 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.346535 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.346749 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.346708 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.346857 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.346838 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.346922 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.346882 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.347260 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.347240 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.348643 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.348626 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.348923 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.348905 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.373219 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.373192 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.373394 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.373371 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lfq6\" (UniqueName: \"kubernetes.io/projected/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-kube-api-access-5lfq6\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.479114 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.479048 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:01.816254 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.816216 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm"] Oct 30 18:05:01.830957 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:05:01.830926 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8b0a7a_6aef_4237_acb3_c9e2ee5112fd.slice/crio-361fbc1b708fddbe1b9cabbd02f27882ca2c97b437c805fe699b8e0408484e60 WatchSource:0}: Error finding container 361fbc1b708fddbe1b9cabbd02f27882ca2c97b437c805fe699b8e0408484e60: Status 404 returned error can't find the container with id 361fbc1b708fddbe1b9cabbd02f27882ca2c97b437c805fe699b8e0408484e60 Oct 30 18:05:01.834880 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.834836 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:05:01.835093 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.834932 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:05:01.835093 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.834973 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:05:01.840602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:01.840576 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:02.345979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:02.345940 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" event={"ID":"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd","Type":"ContainerStarted","Data":"0e8affd8ce1ebaa1693512f13563745530ee656c8354929f2932d6fde0ca3fb0"} Oct 30 18:05:02.346593 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:02.346566 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" event={"ID":"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd","Type":"ContainerStarted","Data":"361fbc1b708fddbe1b9cabbd02f27882ca2c97b437c805fe699b8e0408484e60"} Oct 30 18:05:02.423355 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:02.423290 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" podStartSLOduration=1.42327145 podStartE2EDuration="1.42327145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:02.409016703 +0000 UTC m=+763.124586588" watchObservedRunningTime="2026-10-30 18:05:02.42327145 +0000 UTC m=+763.138841325" Oct 30 18:05:02.479336 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:02.479304 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:02.480744 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:02.480724 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 18:05:02.480805 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:02.480766 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" podUID="db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 18:05:03.479561 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:03.479486 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 18:05:03.480012 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:03.479586 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" podUID="db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 18:05:03.649525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:03.648670 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:04.479389 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:04.479350 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 18:05:04.479619 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:04.479406 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" podUID="db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 18:05:05.387735 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:05.387702 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:05.480438 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:05.480403 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 18:05:05.480932 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:05.480461 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" podUID="db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 18:05:06.479592 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:06.479556 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 18:05:06.479807 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:06.479615 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" podUID="db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 18:05:07.171719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:07.168272 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:07.479879 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:07.479808 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 18:05:07.479879 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:07.479861 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" podUID="db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 18:05:07.609526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:07.608558 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:05:07.626392 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:07.626371 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:05:08.409014 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:08.408981 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:05:09.016573 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:09.010655 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:09.228681 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:09.228648 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:09.229351 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:09.229329 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:09.230235 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:09.230215 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:05:09.294903 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:09.294869 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:10.809934 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:10.803687 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:12.485566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:12.483215 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:12.904373 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:12.904335 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:12.951816 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:12.951790 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:05:13.270395 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.270365 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z"] Oct 30 18:05:13.273202 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.273184 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:13.306537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.304916 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3cc458\"/\"default-dockercfg-tfdlb\"" Oct 30 18:05:13.306537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.304953 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3cc458\"/\"openshift-service-ca.crt\"" Oct 30 18:05:13.306537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.305172 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3cc458\"/\"kube-root-ca.crt\"" Oct 30 18:05:13.306537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.305193 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3cc458\"/\"premium-w3-3cc458-kserve-self-signed-certs\"" Oct 30 18:05:13.307385 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.307356 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z"] Oct 30 18:05:13.329651 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.329627 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-tmp-dir\") pod \"premium-w3-3cc458-kserve-6554499db7-7zf2z\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:13.329737 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.329675 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-model-cache\") pod \"premium-w3-3cc458-kserve-6554499db7-7zf2z\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:13.329784 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.329734 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-home\") pod \"premium-w3-3cc458-kserve-6554499db7-7zf2z\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:13.329848 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.329834 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-dshm\") pod \"premium-w3-3cc458-kserve-6554499db7-7zf2z\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:13.329884 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.329857 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88mgj\" (UniqueName: \"kubernetes.io/projected/2d8dda47-f665-4fcb-bdae-3430e419841c-kube-api-access-88mgj\") pod \"premium-w3-3cc458-kserve-6554499db7-7zf2z\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:13.329966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.329955 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d8dda47-f665-4fcb-bdae-3430e419841c-tls-certs\") pod \"premium-w3-3cc458-kserve-6554499db7-7zf2z\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:13.391200 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.391167 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:13.430717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.430698 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-dshm\") pod \"premium-w3-3cc458-kserve-6554499db7-7zf2z\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:13.430844 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.430722 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88mgj\" (UniqueName: \"kubernetes.io/projected/2d8dda47-f665-4fcb-bdae-3430e419841c-kube-api-access-88mgj\") pod \"premium-w3-3cc458-kserve-6554499db7-7zf2z\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:13.430960 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.430931 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d8dda47-f665-4fcb-bdae-3430e419841c-tls-certs\") pod \"premium-w3-3cc458-kserve-6554499db7-7zf2z\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:13.431145 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.431126 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-tmp-dir\") pod \"premium-w3-3cc458-kserve-6554499db7-7zf2z\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:13.431239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.431198 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-model-cache\") pod \"premium-w3-3cc458-kserve-6554499db7-7zf2z\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:13.431309 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.431257 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-home\") pod \"premium-w3-3cc458-kserve-6554499db7-7zf2z\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:13.431583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.431558 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-model-cache\") pod \"premium-w3-3cc458-kserve-6554499db7-7zf2z\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:13.431682 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.431602 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-tmp-dir\") pod \"premium-w3-3cc458-kserve-6554499db7-7zf2z\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:13.431682 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.431631 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-home\") pod \"premium-w3-3cc458-kserve-6554499db7-7zf2z\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:13.433788 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.433766 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-dshm\") pod \"premium-w3-3cc458-kserve-6554499db7-7zf2z\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:13.434035 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.434018 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d8dda47-f665-4fcb-bdae-3430e419841c-tls-certs\") pod \"premium-w3-3cc458-kserve-6554499db7-7zf2z\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:13.509357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.509286 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88mgj\" (UniqueName: \"kubernetes.io/projected/2d8dda47-f665-4fcb-bdae-3430e419841c-kube-api-access-88mgj\") pod \"premium-w3-3cc458-kserve-6554499db7-7zf2z\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:13.584712 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.584629 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:13.896349 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:13.896328 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z"] Oct 30 18:05:13.898852 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:05:13.898825 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d8dda47_f665_4fcb_bdae_3430e419841c.slice/crio-8696aa34c47ffce26b25ae19f21a6a9aa9c0c616735feaa6a5bb0da108fd32e6 WatchSource:0}: Error finding container 8696aa34c47ffce26b25ae19f21a6a9aa9c0c616735feaa6a5bb0da108fd32e6: Status 404 returned error can't find the container with id 8696aa34c47ffce26b25ae19f21a6a9aa9c0c616735feaa6a5bb0da108fd32e6 Oct 30 18:05:14.433514 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:14.433459 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" event={"ID":"2d8dda47-f665-4fcb-bdae-3430e419841c","Type":"ContainerStarted","Data":"f31f9568e89352f44f7d7b8c70b7eaca385631b7c58ced41b612b1a3a1e25730"} Oct 30 18:05:14.433690 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:14.433539 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" event={"ID":"2d8dda47-f665-4fcb-bdae-3430e419841c","Type":"ContainerStarted","Data":"8696aa34c47ffce26b25ae19f21a6a9aa9c0c616735feaa6a5bb0da108fd32e6"} Oct 30 18:05:14.489438 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:14.489389 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podStartSLOduration=1.489375425 podStartE2EDuration="1.489375425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:14.489154683 +0000 UTC m=+775.204724569" watchObservedRunningTime="2026-10-30 18:05:14.489375425 +0000 UTC m=+775.204945298" Oct 30 18:05:14.503555 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:14.503490 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:14.584818 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:14.584786 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:16.390471 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:16.387995 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:18.388424 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:18.383269 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:20.156996 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:20.156941 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:21.930353 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:05:21.930321 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:05:22.032718 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:22.032679 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:23.585689 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:23.585659 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:23.597982 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:23.597957 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:23.890245 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:23.890127 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:24.495312 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:24.495285 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:05:25.441552 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:25.441492 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:25.888431 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:25.888397 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:26.786943 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:26.786904 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:05:27.912280 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:27.912241 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:28.491831 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.491796 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv"] Oct 30 18:05:28.532864 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.532786 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv"] Oct 30 18:05:28.533193 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.533173 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.659222 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.659183 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.659493 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.659470 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.659717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.659696 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.659861 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.659844 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.659997 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.659980 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.660154 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.660138 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.660269 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.660254 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp9gf\" (UniqueName: \"kubernetes.io/projected/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-kube-api-access-qp9gf\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.660421 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.660404 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.660617 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.660598 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.761950 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.761858 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.761950 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.761899 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.761950 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.761924 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.762242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.761969 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.762242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.761985 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp9gf\" (UniqueName: \"kubernetes.io/projected/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-kube-api-access-qp9gf\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.762242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.762036 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.762242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.762097 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.762242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.762159 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.762242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.762183 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.762591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.762426 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.762591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.762534 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.762591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.762554 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.762734 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.762593 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.762908 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.762879 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.765333 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.765318 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.765574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.765558 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.788171 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.788147 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.788490 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.788475 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp9gf\" (UniqueName: \"kubernetes.io/projected/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-kube-api-access-qp9gf\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:28.908814 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:28.908778 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:29.306579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:29.306549 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv"] Oct 30 18:05:29.307820 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:05:29.307781 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b7d3ed_4e42_4eb6_abfd_ae968273d8b5.slice/crio-cf1280490bdcca624f1ad6dcf5504a4c321f645db2ee9932f33988affb14c367 WatchSource:0}: Error finding container cf1280490bdcca624f1ad6dcf5504a4c321f645db2ee9932f33988affb14c367: Status 404 returned error can't find the container with id cf1280490bdcca624f1ad6dcf5504a4c321f645db2ee9932f33988affb14c367 Oct 30 18:05:29.310263 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:29.310231 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:05:29.310343 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:29.310297 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:05:29.310343 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:29.310328 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:05:29.567662 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:29.567550 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" event={"ID":"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5","Type":"ContainerStarted","Data":"e4858c1f49fde3ecb3fcbc8a2e63832ecbbdc6ce466a3e4a2a8f0919fdd12eeb"} Oct 30 18:05:29.567662 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:29.567602 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" event={"ID":"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5","Type":"ContainerStarted","Data":"cf1280490bdcca624f1ad6dcf5504a4c321f645db2ee9932f33988affb14c367"} Oct 30 18:05:29.615126 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:29.615079 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" podStartSLOduration=1.61506577 podStartE2EDuration="1.61506577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:29.614594421 +0000 UTC m=+790.330164296" watchObservedRunningTime="2026-10-30 18:05:29.61506577 +0000 UTC m=+790.330635640" Oct 30 18:05:29.878472 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:29.874638 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:29.909236 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:29.909196 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:30.913659 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:30.913625 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:05:30.914152 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:30.913694 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" podUID="d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 30 18:05:31.706528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:31.697933 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:31.911334 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:31.910891 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:05:31.911334 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:31.910951 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" podUID="d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 30 18:05:32.913952 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:32.913915 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:05:32.914478 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:32.914031 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" podUID="d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 30 18:05:33.647952 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:33.647920 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:33.911587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:33.910873 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:05:33.911587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:33.910936 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" podUID="d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 30 18:05:34.910980 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:34.910947 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:05:34.911607 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:34.911566 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" podUID="d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 30 18:05:35.429466 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:35.429420 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:35.909527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:35.909124 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:05:35.909527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:35.909182 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" podUID="d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 30 18:05:36.913222 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:36.913192 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:36.913694 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:36.913461 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:36.914243 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:36.914228 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:05:37.301925 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:37.301892 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:38.972041 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:38.972003 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8"] Oct 30 18:05:38.974752 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:38.974731 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:38.999938 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:38.999879 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-57ee15\"/\"premium-w0-57ee15-kserve-self-signed-certs\"" Oct 30 18:05:39.011735 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.011710 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-57ee15\"/\"openshift-service-ca.crt\"" Oct 30 18:05:39.011735 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.011721 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-57ee15\"/\"default-dockercfg-d54vq\"" Oct 30 18:05:39.011945 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.011928 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-57ee15\"/\"kube-root-ca.crt\"" Oct 30 18:05:39.014160 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.014142 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8"] Oct 30 18:05:39.015353 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.015336 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcebcb0f-0443-4e43-9850-e4bb46d279f4-tls-certs\") pod \"premium-w0-57ee15-kserve-7c85cdb46-h95c8\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:39.015476 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.015453 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzswl\" (UniqueName: \"kubernetes.io/projected/dcebcb0f-0443-4e43-9850-e4bb46d279f4-kube-api-access-tzswl\") pod \"premium-w0-57ee15-kserve-7c85cdb46-h95c8\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:39.015535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.015493 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-dshm\") pod \"premium-w0-57ee15-kserve-7c85cdb46-h95c8\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:39.015598 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.015582 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-model-cache\") pod \"premium-w0-57ee15-kserve-7c85cdb46-h95c8\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:39.015644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.015631 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-tmp-dir\") pod \"premium-w0-57ee15-kserve-7c85cdb46-h95c8\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:39.015689 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.015652 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-home\") pod \"premium-w0-57ee15-kserve-7c85cdb46-h95c8\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:39.116412 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.116382 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzswl\" (UniqueName: \"kubernetes.io/projected/dcebcb0f-0443-4e43-9850-e4bb46d279f4-kube-api-access-tzswl\") pod \"premium-w0-57ee15-kserve-7c85cdb46-h95c8\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:39.116594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.116419 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-dshm\") pod \"premium-w0-57ee15-kserve-7c85cdb46-h95c8\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:39.116594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.116472 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-model-cache\") pod \"premium-w0-57ee15-kserve-7c85cdb46-h95c8\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:39.116594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.116493 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-tmp-dir\") pod \"premium-w0-57ee15-kserve-7c85cdb46-h95c8\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:39.116594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.116533 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-home\") pod \"premium-w0-57ee15-kserve-7c85cdb46-h95c8\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:39.116810 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.116595 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcebcb0f-0443-4e43-9850-e4bb46d279f4-tls-certs\") pod \"premium-w0-57ee15-kserve-7c85cdb46-h95c8\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:39.116916 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.116888 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-model-cache\") pod \"premium-w0-57ee15-kserve-7c85cdb46-h95c8\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:39.117023 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.116954 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-tmp-dir\") pod \"premium-w0-57ee15-kserve-7c85cdb46-h95c8\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:39.117023 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.116988 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-home\") pod \"premium-w0-57ee15-kserve-7c85cdb46-h95c8\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:39.119007 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.118985 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-dshm\") pod \"premium-w0-57ee15-kserve-7c85cdb46-h95c8\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:39.119210 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.119194 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcebcb0f-0443-4e43-9850-e4bb46d279f4-tls-certs\") pod \"premium-w0-57ee15-kserve-7c85cdb46-h95c8\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:39.139792 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.139760 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:39.140457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.140438 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzswl\" (UniqueName: \"kubernetes.io/projected/dcebcb0f-0443-4e43-9850-e4bb46d279f4-kube-api-access-tzswl\") pod \"premium-w0-57ee15-kserve-7c85cdb46-h95c8\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:39.287232 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.287201 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:39.318776 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.318748 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:39.597519 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.597480 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8"] Oct 30 18:05:39.599124 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:05:39.599097 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcebcb0f_0443_4e43_9850_e4bb46d279f4.slice/crio-0b59fadd4b5571000b30d091411edb1a64b173ac7c0de8ed16e224a42f83a590 WatchSource:0}: Error finding container 0b59fadd4b5571000b30d091411edb1a64b173ac7c0de8ed16e224a42f83a590: Status 404 returned error can't find the container with id 0b59fadd4b5571000b30d091411edb1a64b173ac7c0de8ed16e224a42f83a590 Oct 30 18:05:39.644345 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:39.644318 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" event={"ID":"dcebcb0f-0443-4e43-9850-e4bb46d279f4","Type":"ContainerStarted","Data":"0b59fadd4b5571000b30d091411edb1a64b173ac7c0de8ed16e224a42f83a590"} Oct 30 18:05:40.652084 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:40.652050 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" event={"ID":"dcebcb0f-0443-4e43-9850-e4bb46d279f4","Type":"ContainerStarted","Data":"e52baa7cb72887c1a9f1cf8e43d9bdf085371d91e39f1ea9f314ef3884e8a2f8"} Oct 30 18:05:40.698153 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:40.698107 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podStartSLOduration=2.698093443 podStartE2EDuration="2.698093443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:40.696969435 +0000 UTC m=+801.412539308" watchObservedRunningTime="2026-10-30 18:05:40.698093443 +0000 UTC m=+801.413663318" Oct 30 18:05:41.010590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:41.006432 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:41.287472 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:41.287446 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:43.100894 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:43.097402 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:45.326546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:45.324158 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:46.296261 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.296234 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:05:46.318682 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.318654 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw"] Oct 30 18:05:46.321195 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.321180 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.358959 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.358925 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw"] Oct 30 18:05:46.431026 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.430937 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scvgj\" (UniqueName: \"kubernetes.io/projected/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-kube-api-access-scvgj\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.431237 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.431038 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.431237 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.431074 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.431237 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.431094 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.431237 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.431191 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.431482 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.431257 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.431482 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.431410 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.431482 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.431446 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.431697 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.431536 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.532696 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.532664 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scvgj\" (UniqueName: \"kubernetes.io/projected/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-kube-api-access-scvgj\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.532837 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.532707 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.532837 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.532727 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.532972 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.532849 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.532972 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.532925 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.533072 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.532990 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.533122 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.533101 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.533180 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.533132 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.533235 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.533190 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.533235 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.533130 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.533498 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.533475 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.533609 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.533541 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.533788 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.533764 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.533842 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.533802 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.535283 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.535255 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.535629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.535612 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.556516 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.556439 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.556608 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.556537 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scvgj\" (UniqueName: \"kubernetes.io/projected/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-kube-api-access-scvgj\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.634049 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.634019 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:46.967153 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.967124 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw"] Oct 30 18:05:46.968734 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:05:46.968706 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47cefcfc_ddae_4c02_ad2c_ae2cb8769e64.slice/crio-26a5a98cefa4f2f963d18e548596e3471768c80e1b172378b5ec1618205b3059 WatchSource:0}: Error finding container 26a5a98cefa4f2f963d18e548596e3471768c80e1b172378b5ec1618205b3059: Status 404 returned error can't find the container with id 26a5a98cefa4f2f963d18e548596e3471768c80e1b172378b5ec1618205b3059 Oct 30 18:05:46.971108 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.971075 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:05:46.971212 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.971137 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:05:46.971212 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:46.971168 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:05:47.150370 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:47.150268 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:47.711528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:47.711452 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" event={"ID":"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64","Type":"ContainerStarted","Data":"88a9eaba348c52a304694c657c04c62b48cd9d534a7106e34e43a3fe117a1905"} Oct 30 18:05:47.712067 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:47.711744 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" event={"ID":"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64","Type":"ContainerStarted","Data":"26a5a98cefa4f2f963d18e548596e3471768c80e1b172378b5ec1618205b3059"} Oct 30 18:05:48.634556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:48.634521 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:48.635902 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:48.635878 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:05:48.636027 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:48.635922 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" podUID="47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 18:05:48.889442 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:48.889358 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:49.288134 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:49.288104 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:49.322876 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:49.322851 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:49.367370 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:49.367322 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" podStartSLOduration=3.367308015 podStartE2EDuration="3.367308015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:47.779915097 +0000 UTC m=+808.495484971" watchObservedRunningTime="2026-10-30 18:05:49.367308015 +0000 UTC m=+810.082877897" Oct 30 18:05:49.635009 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:49.634927 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:05:49.635009 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:49.634982 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" podUID="47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 18:05:49.734549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:49.734521 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:05:50.635173 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:50.635140 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:05:50.635598 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:50.635213 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" podUID="47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 18:05:50.683554 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:50.683369 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:51.634445 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:51.634414 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:05:51.634638 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:51.634464 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" podUID="47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 18:05:51.938167 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:05:51.938078 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:05:52.425524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:52.418701 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:52.634856 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:52.634824 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:05:52.635099 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:52.634874 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" podUID="47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 18:05:53.783243 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:53.783214 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:53.783781 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:53.783526 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:53.784726 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:53.784703 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:05:54.191444 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:54.191337 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:56.098547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:56.098356 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:57.877282 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:57.877242 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:59.679628 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:05:59.676238 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:01.372675 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:01.372636 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:02.261129 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:02.261096 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:06:03.257533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:03.255288 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:05.363656 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:05.363621 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:07.214047 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:07.213911 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:08.873330 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:08.873274 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:10.549429 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:10.549395 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:12.253913 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:12.253868 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:12.992840 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:12.992804 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:13.955378 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:13.955338 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:14.472023 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:14.471992 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:14.611675 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:14.611652 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:15.636527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:15.616331 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:17.279240 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:17.279202 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:18.952812 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:18.952777 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:20.682247 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:20.681378 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:21.929681 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:06:21.929650 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:06:22.319718 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:22.319683 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:24.086711 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.086240 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:24.130001 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.129956 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j"] Oct 30 18:06:24.136268 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.136227 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.158978 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.158949 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j"] Oct 30 18:06:24.289181 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.289142 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.289340 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.289205 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.289340 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.289272 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg4nr\" (UniqueName: \"kubernetes.io/projected/ec3c95aa-215c-4f15-8dc8-bc0beb954236-kube-api-access-gg4nr\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.289340 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.289311 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.289340 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.289326 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.289494 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.289432 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.289494 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.289456 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.289494 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.289493 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.289691 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.289655 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.390991 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.390923 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.390991 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.390955 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.391177 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.390993 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.391177 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.391018 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.391177 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.391051 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.391177 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.391098 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.391177 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.391153 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.391427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.391210 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.391427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.391353 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.391427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.391403 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg4nr\" (UniqueName: \"kubernetes.io/projected/ec3c95aa-215c-4f15-8dc8-bc0beb954236-kube-api-access-gg4nr\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.391427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.391415 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.391770 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.391460 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.391770 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.391686 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.392126 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.392104 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.393652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.393626 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.393781 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.393763 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.413661 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.413638 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.413995 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.413973 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg4nr\" (UniqueName: \"kubernetes.io/projected/ec3c95aa-215c-4f15-8dc8-bc0beb954236-kube-api-access-gg4nr\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.452997 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.452978 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:24.803308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.803281 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j"] Oct 30 18:06:24.804816 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:06:24.804792 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec3c95aa_215c_4f15_8dc8_bc0beb954236.slice/crio-973cb30f59f0578a083423919b6334e606a7d024744357145f7f1c1887b7b911 WatchSource:0}: Error finding container 973cb30f59f0578a083423919b6334e606a7d024744357145f7f1c1887b7b911: Status 404 returned error can't find the container with id 973cb30f59f0578a083423919b6334e606a7d024744357145f7f1c1887b7b911 Oct 30 18:06:24.807033 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.807001 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:06:24.807126 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.807067 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:06:24.807126 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.807094 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:06:24.971410 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.971377 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" event={"ID":"ec3c95aa-215c-4f15-8dc8-bc0beb954236","Type":"ContainerStarted","Data":"6d25961fc4da23bdd7bf8c4c12a2971d7f2498c46d49eb1ace177ff767db29ce"} Oct 30 18:06:24.971619 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:24.971427 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" event={"ID":"ec3c95aa-215c-4f15-8dc8-bc0beb954236","Type":"ContainerStarted","Data":"973cb30f59f0578a083423919b6334e606a7d024744357145f7f1c1887b7b911"} Oct 30 18:06:25.015662 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:25.015604 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" podStartSLOduration=1.015586867 podStartE2EDuration="1.015586867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:25.015129605 +0000 UTC m=+845.730699482" watchObservedRunningTime="2026-10-30 18:06:25.015586867 +0000 UTC m=+845.731156744" Oct 30 18:06:25.453265 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:25.453237 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:25.455017 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:25.454988 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:06:25.455130 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:25.455036 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" podUID="ec3c95aa-215c-4f15-8dc8-bc0beb954236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 18:06:25.841428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:25.841393 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:26.453387 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:26.453352 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:06:26.453790 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:26.453404 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" podUID="ec3c95aa-215c-4f15-8dc8-bc0beb954236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 18:06:27.453554 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:27.453521 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:06:27.453947 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:27.453571 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" podUID="ec3c95aa-215c-4f15-8dc8-bc0beb954236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 18:06:27.592530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:27.589834 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:28.453510 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.453477 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:06:28.453674 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.453566 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" podUID="ec3c95aa-215c-4f15-8dc8-bc0beb954236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 18:06:28.687867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.687836 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc"] Oct 30 18:06:28.713827 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.713754 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.718313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.718121 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc"] Oct 30 18:06:28.857869 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.857830 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7882513-ef6d-4b11-871e-39a29ad022df-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.858049 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.857877 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.858049 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.857968 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.858049 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.857985 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7882513-ef6d-4b11-871e-39a29ad022df-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.858049 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.858007 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.858049 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.858043 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r6d9\" (UniqueName: \"kubernetes.io/projected/f7882513-ef6d-4b11-871e-39a29ad022df-kube-api-access-8r6d9\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.858216 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.858136 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.858216 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.858170 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7882513-ef6d-4b11-871e-39a29ad022df-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.858216 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.858189 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.959402 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.959372 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.959611 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.959419 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7882513-ef6d-4b11-871e-39a29ad022df-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.959611 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.959438 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.959611 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.959491 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7882513-ef6d-4b11-871e-39a29ad022df-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.959611 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.959537 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.959611 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.959596 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.959611 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.959613 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7882513-ef6d-4b11-871e-39a29ad022df-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.959921 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.959634 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.959921 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.959672 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r6d9\" (UniqueName: \"kubernetes.io/projected/f7882513-ef6d-4b11-871e-39a29ad022df-kube-api-access-8r6d9\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.959921 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.959891 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.960071 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.959947 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.960108 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.960066 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.960170 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.960146 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.960282 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.960253 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7882513-ef6d-4b11-871e-39a29ad022df-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.961976 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.961959 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.962250 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.962234 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7882513-ef6d-4b11-871e-39a29ad022df-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.982286 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.982231 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7882513-ef6d-4b11-871e-39a29ad022df-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:28.982377 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:28.982302 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r6d9\" (UniqueName: \"kubernetes.io/projected/f7882513-ef6d-4b11-871e-39a29ad022df-kube-api-access-8r6d9\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:29.034518 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:29.034475 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:29.163027 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:29.162990 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:29.365015 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:29.364994 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc"] Oct 30 18:06:29.367290 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:06:29.367269 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7882513_ef6d_4b11_871e_39a29ad022df.slice/crio-4f046068bb8e00146eb9331ed2895a7a39d3ea2480833e1c3f7f280e3a9ccf25 WatchSource:0}: Error finding container 4f046068bb8e00146eb9331ed2895a7a39d3ea2480833e1c3f7f280e3a9ccf25: Status 404 returned error can't find the container with id 4f046068bb8e00146eb9331ed2895a7a39d3ea2480833e1c3f7f280e3a9ccf25 Oct 30 18:06:29.369571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:29.369540 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:06:29.369644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:29.369628 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:06:29.369685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:29.369658 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:06:29.454153 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:29.454133 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:06:29.454531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:29.454175 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" podUID="ec3c95aa-215c-4f15-8dc8-bc0beb954236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 18:06:30.038121 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:30.038081 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" event={"ID":"f7882513-ef6d-4b11-871e-39a29ad022df","Type":"ContainerStarted","Data":"9098652c215ffeb76c0a5f8ae2d3f9069fbb61ceb96ec93ee5adcb36dc2a401d"} Oct 30 18:06:30.038400 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:30.038382 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" event={"ID":"f7882513-ef6d-4b11-871e-39a29ad022df","Type":"ContainerStarted","Data":"4f046068bb8e00146eb9331ed2895a7a39d3ea2480833e1c3f7f280e3a9ccf25"} Oct 30 18:06:30.117199 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:30.117133 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" podStartSLOduration=2.117113279 podStartE2EDuration="2.117113279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:30.095903856 +0000 UTC m=+850.811473732" watchObservedRunningTime="2026-10-30 18:06:30.117113279 +0000 UTC m=+850.832683155" Oct 30 18:06:30.453927 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:30.453846 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:06:30.453927 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:30.453898 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" podUID="ec3c95aa-215c-4f15-8dc8-bc0beb954236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 18:06:30.970796 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:30.970734 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:31.040911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:31.040880 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:31.043692 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:31.043658 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:06:31.043843 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:31.043720 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" podUID="f7882513-ef6d-4b11-871e-39a29ad022df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 18:06:31.454231 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:31.454197 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:06:31.454444 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:31.454273 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" podUID="ec3c95aa-215c-4f15-8dc8-bc0beb954236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 18:06:32.034987 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:32.034951 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:06:32.035394 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:32.035008 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" podUID="f7882513-ef6d-4b11-871e-39a29ad022df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 18:06:32.457264 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:32.457156 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:32.457563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:32.457542 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:32.458289 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:32.458270 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:06:32.761278 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:32.761179 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:33.034960 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:33.034923 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:06:33.035157 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:33.034983 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" podUID="f7882513-ef6d-4b11-871e-39a29ad022df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 18:06:34.035682 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.035650 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:06:34.036085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.035714 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" podUID="f7882513-ef6d-4b11-871e-39a29ad022df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 18:06:34.329635 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.329556 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q"] Oct 30 18:06:34.333529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.333514 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.363320 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.363298 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q"] Oct 30 18:06:34.458657 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.458626 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.458831 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.458671 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f49829-73ed-4760-9444-f1793861eba6-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.458831 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.458709 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f49829-73ed-4760-9444-f1793861eba6-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.458831 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.458798 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f49829-73ed-4760-9444-f1793861eba6-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.458831 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.458828 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.459002 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.458849 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.459002 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.458965 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.459116 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.459053 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.459116 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.459071 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lstdn\" (UniqueName: \"kubernetes.io/projected/81f49829-73ed-4760-9444-f1793861eba6-kube-api-access-lstdn\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.560375 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.560341 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.560375 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.560381 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f49829-73ed-4760-9444-f1793861eba6-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.560673 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.560404 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f49829-73ed-4760-9444-f1793861eba6-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.560673 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.560437 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f49829-73ed-4760-9444-f1793861eba6-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.560673 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.560459 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.560673 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.560476 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.560673 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.560546 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.560927 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.560700 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.560927 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.560726 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lstdn\" (UniqueName: \"kubernetes.io/projected/81f49829-73ed-4760-9444-f1793861eba6-kube-api-access-lstdn\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.560927 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.560830 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.561190 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.561161 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.561298 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.561215 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.561543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.561293 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.561543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.561388 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f49829-73ed-4760-9444-f1793861eba6-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.563258 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.563235 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f49829-73ed-4760-9444-f1793861eba6-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.563258 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.563254 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.586045 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.585990 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lstdn\" (UniqueName: \"kubernetes.io/projected/81f49829-73ed-4760-9444-f1793861eba6-kube-api-access-lstdn\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.586311 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.586290 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f49829-73ed-4760-9444-f1793861eba6-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:34.643959 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.643928 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:34.646820 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:34.646794 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:35.034854 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:35.034825 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:06:35.035011 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:35.034876 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" podUID="f7882513-ef6d-4b11-871e-39a29ad022df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 18:06:35.081255 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:35.081232 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q"] Oct 30 18:06:35.083854 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:06:35.083831 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f49829_73ed_4760_9444_f1793861eba6.slice/crio-b6c444c8f6219d6c305ec47e6a84b2e4e1fe586222b1df7312fac134e428fc9f WatchSource:0}: Error finding container b6c444c8f6219d6c305ec47e6a84b2e4e1fe586222b1df7312fac134e428fc9f: Status 404 returned error can't find the container with id b6c444c8f6219d6c305ec47e6a84b2e4e1fe586222b1df7312fac134e428fc9f Oct 30 18:06:35.086111 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:35.086085 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:06:35.086177 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:35.086146 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:06:35.086217 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:35.086174 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:06:36.035354 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:36.035317 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:06:36.035560 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:36.035386 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" podUID="f7882513-ef6d-4b11-871e-39a29ad022df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 18:06:36.078678 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:36.078643 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" event={"ID":"81f49829-73ed-4760-9444-f1793861eba6","Type":"ContainerStarted","Data":"9c432aa616a5818288b00b8aef7e0e4736137daa708271abf88196d24d5811a8"} Oct 30 18:06:36.078887 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:36.078702 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" event={"ID":"81f49829-73ed-4760-9444-f1793861eba6","Type":"ContainerStarted","Data":"b6c444c8f6219d6c305ec47e6a84b2e4e1fe586222b1df7312fac134e428fc9f"} Oct 30 18:06:36.153306 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:36.153228 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" podStartSLOduration=2.1532094649999998 podStartE2EDuration="2.153209465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:36.1374603 +0000 UTC m=+856.853030174" watchObservedRunningTime="2026-10-30 18:06:36.153209465 +0000 UTC m=+856.868779340" Oct 30 18:06:36.532186 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:36.532151 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:36.647887 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:36.647855 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:36.649477 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:36.649453 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 18:06:36.649601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:36.649497 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" podUID="81f49829-73ed-4760-9444-f1793861eba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 18:06:37.039082 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:37.039057 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:37.039518 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:37.039481 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:37.040680 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:37.040662 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:06:37.650916 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:37.650879 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 18:06:37.651533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:37.651484 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" podUID="81f49829-73ed-4760-9444-f1793861eba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 18:06:38.650529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:38.647632 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 18:06:38.650529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:38.647702 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" podUID="81f49829-73ed-4760-9444-f1793861eba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 18:06:39.086239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:39.086201 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:39.611721 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:39.611662 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:06:39.651973 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:39.651942 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 18:06:39.652269 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:39.652226 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" podUID="81f49829-73ed-4760-9444-f1793861eba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 18:06:40.349529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:40.347427 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:06:40.349529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:40.347832 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Oct 30 18:06:40.398860 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:40.398734 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:06:40.618468 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:40.618347 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:06:40.619304 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:40.619281 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:06:40.648842 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:40.648810 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 18:06:40.649097 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:40.649070 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" podUID="81f49829-73ed-4760-9444-f1793861eba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 18:06:40.756550 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:40.756466 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:06:41.141128 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:41.141092 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:41.647706 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:41.647677 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 18:06:41.648092 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:41.647728 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" podUID="81f49829-73ed-4760-9444-f1793861eba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 18:06:42.648021 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:42.647976 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 18:06:42.648428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:42.648047 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" podUID="81f49829-73ed-4760-9444-f1793861eba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 18:06:42.802678 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:42.802641 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:43.809651 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:43.809618 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:43.810058 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:43.809942 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:43.810918 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:43.810900 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:06:44.394473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:44.394440 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:45.428458 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:45.428427 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:45.574012 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:45.573973 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:46.233357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:46.233325 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:46.941044 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:46.941005 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:47.171760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.171723 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-69644977dd-gvwzm"] Oct 30 18:06:47.175761 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.175738 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" Oct 30 18:06:47.204753 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.204692 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-25c3ad6fd807\"" Oct 30 18:06:47.204924 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.204700 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 18:06:47.204924 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.204713 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 18:06:47.205070 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.204715 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 18:06:47.205070 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.204729 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 18:06:47.205070 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.204806 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-24576edb8e27\"" Oct 30 18:06:47.205070 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.204910 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-hzm64\"" Oct 30 18:06:47.207289 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.207269 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-69644977dd-gvwzm"] Oct 30 18:06:47.292526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.289296 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94a706c7-905a-4868-ada3-6d2f576829fa-provider-credentials\") pod \"payload-processing-external-model-69644977dd-gvwzm\" (UID: \"94a706c7-905a-4868-ada3-6d2f576829fa\") " pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" Oct 30 18:06:47.292526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.289381 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94a706c7-905a-4868-ada3-6d2f576829fa-routing-overlay-volume\") pod \"payload-processing-external-model-69644977dd-gvwzm\" (UID: \"94a706c7-905a-4868-ada3-6d2f576829fa\") " pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" Oct 30 18:06:47.292526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.289417 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a706c7-905a-4868-ada3-6d2f576829fa-plugins-config-volume\") pod \"payload-processing-external-model-69644977dd-gvwzm\" (UID: \"94a706c7-905a-4868-ada3-6d2f576829fa\") " pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" Oct 30 18:06:47.300560 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.297651 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh7xf\" (UniqueName: \"kubernetes.io/projected/94a706c7-905a-4868-ada3-6d2f576829fa-kube-api-access-qh7xf\") pod \"payload-processing-external-model-69644977dd-gvwzm\" (UID: \"94a706c7-905a-4868-ada3-6d2f576829fa\") " pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" Oct 30 18:06:47.410526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.409538 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94a706c7-905a-4868-ada3-6d2f576829fa-provider-credentials\") pod \"payload-processing-external-model-69644977dd-gvwzm\" (UID: \"94a706c7-905a-4868-ada3-6d2f576829fa\") " pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" Oct 30 18:06:47.410526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.409644 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94a706c7-905a-4868-ada3-6d2f576829fa-routing-overlay-volume\") pod \"payload-processing-external-model-69644977dd-gvwzm\" (UID: \"94a706c7-905a-4868-ada3-6d2f576829fa\") " pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" Oct 30 18:06:47.410526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.409697 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a706c7-905a-4868-ada3-6d2f576829fa-plugins-config-volume\") pod \"payload-processing-external-model-69644977dd-gvwzm\" (UID: \"94a706c7-905a-4868-ada3-6d2f576829fa\") " pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" Oct 30 18:06:47.410526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.409849 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh7xf\" (UniqueName: \"kubernetes.io/projected/94a706c7-905a-4868-ada3-6d2f576829fa-kube-api-access-qh7xf\") pod \"payload-processing-external-model-69644977dd-gvwzm\" (UID: \"94a706c7-905a-4868-ada3-6d2f576829fa\") " pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" Oct 30 18:06:47.411266 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.411235 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94a706c7-905a-4868-ada3-6d2f576829fa-routing-overlay-volume\") pod \"payload-processing-external-model-69644977dd-gvwzm\" (UID: \"94a706c7-905a-4868-ada3-6d2f576829fa\") " pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" Oct 30 18:06:47.412590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.412524 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a706c7-905a-4868-ada3-6d2f576829fa-plugins-config-volume\") pod \"payload-processing-external-model-69644977dd-gvwzm\" (UID: \"94a706c7-905a-4868-ada3-6d2f576829fa\") " pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" Oct 30 18:06:47.417320 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.417264 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94a706c7-905a-4868-ada3-6d2f576829fa-provider-credentials\") pod \"payload-processing-external-model-69644977dd-gvwzm\" (UID: \"94a706c7-905a-4868-ada3-6d2f576829fa\") " pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" Oct 30 18:06:47.439831 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.439809 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh7xf\" (UniqueName: \"kubernetes.io/projected/94a706c7-905a-4868-ada3-6d2f576829fa-kube-api-access-qh7xf\") pod \"payload-processing-external-model-69644977dd-gvwzm\" (UID: \"94a706c7-905a-4868-ada3-6d2f576829fa\") " pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" Oct 30 18:06:47.489678 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.489581 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" Oct 30 18:06:47.818459 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:47.818426 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-69644977dd-gvwzm"] Oct 30 18:06:47.822319 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:06:47.822282 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a706c7_905a_4868_ada3_6d2f576829fa.slice/crio-e62ab2557f7c37aedbdb0b18e9b7dbb3f7be3e29d4454506fb404573dbf9c299 WatchSource:0}: Error finding container e62ab2557f7c37aedbdb0b18e9b7dbb3f7be3e29d4454506fb404573dbf9c299: Status 404 returned error can't find the container with id e62ab2557f7c37aedbdb0b18e9b7dbb3f7be3e29d4454506fb404573dbf9c299 Oct 30 18:06:48.065531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:48.060444 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:48.251605 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:48.251492 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" event={"ID":"94a706c7-905a-4868-ada3-6d2f576829fa","Type":"ContainerStarted","Data":"421bad578b7d43225d74fac5fe7c8bafeaa2f833ab9072ca22d73086f8327ff5"} Oct 30 18:06:48.251605 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:48.251573 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" Oct 30 18:06:48.251605 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:48.251595 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" event={"ID":"94a706c7-905a-4868-ada3-6d2f576829fa","Type":"ContainerStarted","Data":"e62ab2557f7c37aedbdb0b18e9b7dbb3f7be3e29d4454506fb404573dbf9c299"} Oct 30 18:06:48.293569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:48.293483 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" podStartSLOduration=1.293468322 podStartE2EDuration="1.293468322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:48.291877561 +0000 UTC m=+869.007447434" watchObservedRunningTime="2026-10-30 18:06:48.293468322 +0000 UTC m=+869.009038196" Oct 30 18:06:49.865418 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:49.865372 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:51.525682 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:51.525643 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:51.930373 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:06:51.930291 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:06:53.275911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:53.275881 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:55.004522 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:55.004466 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:56.790713 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:56.790659 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:58.453875 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:58.453843 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:59.262344 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:06:59.262319 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" Oct 30 18:07:00.139094 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:00.139039 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:01.848181 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:01.848140 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:03.556598 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:03.556567 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:04.434380 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.434348 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj"] Oct 30 18:07:04.439470 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.439450 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.466774 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.466749 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj"] Oct 30 18:07:04.536753 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.536724 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7j8g\" (UniqueName: \"kubernetes.io/projected/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-kube-api-access-t7j8g\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.536917 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.536780 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.536917 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.536868 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.536917 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.536901 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.537054 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.536929 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.537054 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.537013 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.537054 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.537031 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.537204 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.537076 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.537258 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.537207 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.637963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.637936 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.638413 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.637974 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.638413 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.638000 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.638413 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.638044 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.638413 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.638062 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.638413 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.638120 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.638413 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.638151 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.638413 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.638197 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7j8g\" (UniqueName: \"kubernetes.io/projected/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-kube-api-access-t7j8g\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.638413 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.638259 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.638888 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.638589 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.638888 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.638797 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.638888 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.638830 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.639039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.638935 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.639120 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.639103 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.641314 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.641289 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.641516 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.641483 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.661294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.661266 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.661294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.661290 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7j8g\" (UniqueName: \"kubernetes.io/projected/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-kube-api-access-t7j8g\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:04.753162 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:04.753112 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:05.081055 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:05.081024 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj"] Oct 30 18:07:05.083145 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:07:05.083123 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571fc9dd_71e3_4fa7_b1af_e74f0a96654d.slice/crio-615a49976fa88e65a8c5bc32a4aa6a060666326464c2f20e9dbc1f49bf24a373 WatchSource:0}: Error finding container 615a49976fa88e65a8c5bc32a4aa6a060666326464c2f20e9dbc1f49bf24a373: Status 404 returned error can't find the container with id 615a49976fa88e65a8c5bc32a4aa6a060666326464c2f20e9dbc1f49bf24a373 Oct 30 18:07:05.085363 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:05.085336 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:07:05.085416 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:05.085400 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:07:05.085465 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:05.085454 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:07:05.284532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:05.280735 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:05.352889 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:05.352794 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" event={"ID":"571fc9dd-71e3-4fa7-b1af-e74f0a96654d","Type":"ContainerStarted","Data":"310cea5c6e5f288464e3d1d1cbb53ab27bab8bc0ed216f3f3761df8f8bc91bc1"} Oct 30 18:07:05.352889 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:05.352856 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" event={"ID":"571fc9dd-71e3-4fa7-b1af-e74f0a96654d","Type":"ContainerStarted","Data":"615a49976fa88e65a8c5bc32a4aa6a060666326464c2f20e9dbc1f49bf24a373"} Oct 30 18:07:05.405671 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:05.405623 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" podStartSLOduration=1.405607949 podStartE2EDuration="1.405607949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:05.405274097 +0000 UTC m=+886.120843970" watchObservedRunningTime="2026-10-30 18:07:05.405607949 +0000 UTC m=+886.121177822" Oct 30 18:07:05.753578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:05.753487 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:06.755586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:06.755556 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:07:06.755975 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:06.755617 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" podUID="571fc9dd-71e3-4fa7-b1af-e74f0a96654d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 18:07:07.068964 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:07.068927 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:07.753911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:07.753872 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:07:07.754082 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:07.753932 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" podUID="571fc9dd-71e3-4fa7-b1af-e74f0a96654d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 18:07:08.747777 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:08.747738 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:08.753841 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:08.753798 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:07:08.753956 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:08.753886 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" podUID="571fc9dd-71e3-4fa7-b1af-e74f0a96654d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 18:07:08.961448 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:08.961411 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:07:09.765541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:09.763032 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:07:09.765541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:09.763127 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" podUID="571fc9dd-71e3-4fa7-b1af-e74f0a96654d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 18:07:09.834176 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:09.833780 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:10.760696 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:10.760619 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:07:10.760696 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:10.760677 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" podUID="571fc9dd-71e3-4fa7-b1af-e74f0a96654d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 18:07:11.603982 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:11.603945 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:11.756999 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:11.756968 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:07:11.757309 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:11.757261 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" podUID="571fc9dd-71e3-4fa7-b1af-e74f0a96654d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 18:07:12.309526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:12.308601 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:07:12.396529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:12.395588 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:07:12.490798 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:12.490755 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:07:12.754583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:12.753880 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:07:12.754583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:12.753963 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" podUID="571fc9dd-71e3-4fa7-b1af-e74f0a96654d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 18:07:12.802657 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:12.802612 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:07:13.287527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:13.286590 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:13.990968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:13.990942 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:13.991565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:13.991346 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:13.992449 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:13.992424 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:07:14.104707 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:14.104671 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:15.701945 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:15.701914 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:17.440772 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:17.440735 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:19.036831 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:19.036795 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:19.505569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:19.505482 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 18:07:19.518109 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:19.518080 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 18:07:20.752897 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:20.752855 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:22.129489 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:07:22.128694 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:07:22.484209 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:22.484124 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:24.172528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.172459 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:24.560227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.560202 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc"] Oct 30 18:07:24.565401 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.565380 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.593536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.593479 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc"] Oct 30 18:07:24.711266 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.711236 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.711421 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.711281 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqrxs\" (UniqueName: \"kubernetes.io/projected/b664f4ca-736e-4cb1-991d-985d1e6aff34-kube-api-access-kqrxs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.711421 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.711304 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.711524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.711442 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.711524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.711474 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.711593 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.711569 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.711627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.711593 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.711690 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.711669 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.711729 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.711704 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b664f4ca-736e-4cb1-991d-985d1e6aff34-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.812631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.812553 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.812631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.812601 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqrxs\" (UniqueName: \"kubernetes.io/projected/b664f4ca-736e-4cb1-991d-985d1e6aff34-kube-api-access-kqrxs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.812792 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.812716 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.812792 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.812772 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.812862 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.812807 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.812898 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.812868 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.812951 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.812901 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.812951 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.812942 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.813047 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.812967 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b664f4ca-736e-4cb1-991d-985d1e6aff34-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.813259 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.813234 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.813375 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.813273 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.813375 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.813305 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.813467 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.813395 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.814008 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.813987 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b664f4ca-736e-4cb1-991d-985d1e6aff34-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.815089 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.815068 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.815297 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.815280 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.835295 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.835276 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqrxs\" (UniqueName: \"kubernetes.io/projected/b664f4ca-736e-4cb1-991d-985d1e6aff34-kube-api-access-kqrxs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.835390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.835296 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:24.879240 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:24.879219 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:25.201236 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:25.201208 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc"] Oct 30 18:07:25.202534 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:07:25.202491 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb664f4ca_736e_4cb1_991d_985d1e6aff34.slice/crio-3b8a9d8fdb4f7aadb86376ede934cd99ff8ed5dfeba06386b6887ddb30ea1199 WatchSource:0}: Error finding container 3b8a9d8fdb4f7aadb86376ede934cd99ff8ed5dfeba06386b6887ddb30ea1199: Status 404 returned error can't find the container with id 3b8a9d8fdb4f7aadb86376ede934cd99ff8ed5dfeba06386b6887ddb30ea1199 Oct 30 18:07:25.204771 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:25.204744 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:07:25.204871 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:25.204805 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:07:25.204871 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:25.204848 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:07:25.568176 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:25.568134 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" event={"ID":"b664f4ca-736e-4cb1-991d-985d1e6aff34","Type":"ContainerStarted","Data":"80d5343a4d7fab6f7441d8100515aaec7fabac590594da9544683a876a70c15d"} Oct 30 18:07:25.568372 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:25.568210 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" event={"ID":"b664f4ca-736e-4cb1-991d-985d1e6aff34","Type":"ContainerStarted","Data":"3b8a9d8fdb4f7aadb86376ede934cd99ff8ed5dfeba06386b6887ddb30ea1199"} Oct 30 18:07:25.614466 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:25.614407 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" podStartSLOduration=1.6143924090000001 podStartE2EDuration="1.614392409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:25.611886136 +0000 UTC m=+906.327456023" watchObservedRunningTime="2026-10-30 18:07:25.614392409 +0000 UTC m=+906.329962284" Oct 30 18:07:25.837231 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:25.837154 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:25.882581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:25.881610 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:26.880794 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:26.880763 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:07:26.881189 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:26.880822 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" podUID="b664f4ca-736e-4cb1-991d-985d1e6aff34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 18:07:27.553547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:27.552847 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:27.884631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:27.880528 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:07:27.884631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:27.880601 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" podUID="b664f4ca-736e-4cb1-991d-985d1e6aff34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 18:07:27.973604 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:27.973566 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:28.880307 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:28.880272 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:07:28.880485 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:28.880334 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" podUID="b664f4ca-736e-4cb1-991d-985d1e6aff34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 18:07:28.953651 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:28.953621 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj"] Oct 30 18:07:28.957717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:28.957701 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:28.975819 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:28.975787 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:28.975987 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:28.975847 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:28.975987 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:28.975873 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:28.975987 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:28.975937 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6hrl\" (UniqueName: \"kubernetes.io/projected/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-kube-api-access-f6hrl\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:28.975987 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:28.975975 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:28.976227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:28.976062 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:28.976227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:28.976162 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:28.976227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:28.976214 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:28.976393 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:28.976239 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:29.020531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.018480 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj"] Oct 30 18:07:29.077006 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.076977 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:29.077162 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.077023 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:29.077162 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.077041 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:29.077162 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.077074 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:29.077162 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.077102 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:29.077162 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.077121 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:29.077414 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.077186 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6hrl\" (UniqueName: \"kubernetes.io/projected/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-kube-api-access-f6hrl\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:29.077414 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.077225 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:29.077414 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.077335 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:29.077616 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.077461 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:29.077681 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.077650 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:29.077766 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.077744 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:29.077823 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.077761 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:29.078048 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.077912 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:29.079597 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.079578 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:29.079815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.079798 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:29.119559 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.117206 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6hrl\" (UniqueName: \"kubernetes.io/projected/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-kube-api-access-f6hrl\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:29.120021 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.120002 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:29.275056 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.275030 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:29.359525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.359404 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:29.630579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.630557 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj"] Oct 30 18:07:29.631998 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:07:29.631974 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e2469b_dc0d_40ab_bd74_a47c4a9e438b.slice/crio-7f6390eee337d37c885cb4441e3136b4d2e6509de52f65d7f1c5fe5b67eb6f07 WatchSource:0}: Error finding container 7f6390eee337d37c885cb4441e3136b4d2e6509de52f65d7f1c5fe5b67eb6f07: Status 404 returned error can't find the container with id 7f6390eee337d37c885cb4441e3136b4d2e6509de52f65d7f1c5fe5b67eb6f07 Oct 30 18:07:29.634279 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.634241 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:07:29.634383 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.634318 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:07:29.634383 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.634369 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:07:29.880393 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.880298 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:07:29.880393 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:29.880356 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" podUID="b664f4ca-736e-4cb1-991d-985d1e6aff34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 18:07:30.642553 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:30.641464 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" event={"ID":"58e2469b-dc0d-40ab-bd74-a47c4a9e438b","Type":"ContainerStarted","Data":"113c96e7649a6ae6d7ea602cfcf4fa522f5e7e5fdf3d1fe4107a9ff9456ea918"} Oct 30 18:07:30.642553 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:30.641544 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" event={"ID":"58e2469b-dc0d-40ab-bd74-a47c4a9e438b","Type":"ContainerStarted","Data":"7f6390eee337d37c885cb4441e3136b4d2e6509de52f65d7f1c5fe5b67eb6f07"} Oct 30 18:07:30.726527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:30.718745 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" podStartSLOduration=2.7187242659999997 podStartE2EDuration="2.718724266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:30.699906129 +0000 UTC m=+911.415476005" watchObservedRunningTime="2026-10-30 18:07:30.718724266 +0000 UTC m=+911.434294141" Oct 30 18:07:30.880566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:30.880533 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:07:30.880713 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:30.880584 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" podUID="b664f4ca-736e-4cb1-991d-985d1e6aff34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 18:07:31.054524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.054474 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:31.275668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.275628 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:31.276842 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.276818 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:07:31.276958 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.276878 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" podUID="58e2469b-dc0d-40ab-bd74-a47c4a9e438b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 18:07:31.620342 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.620305 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc"] Oct 30 18:07:31.625202 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.625175 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.647262 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.644934 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc"] Oct 30 18:07:31.660969 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.660944 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5567046-aa24-421a-8339-fdd9cf56e62c-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.661102 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.661066 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.661170 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.661111 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.661322 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.661298 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.661409 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.661378 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.661471 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.661440 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.661666 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.661644 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.661822 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.661803 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.661954 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.661937 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gmmt\" (UniqueName: \"kubernetes.io/projected/d5567046-aa24-421a-8339-fdd9cf56e62c-kube-api-access-4gmmt\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.763547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.763516 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.763734 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.763565 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.763734 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.763628 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gmmt\" (UniqueName: \"kubernetes.io/projected/d5567046-aa24-421a-8339-fdd9cf56e62c-kube-api-access-4gmmt\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.763734 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.763675 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5567046-aa24-421a-8339-fdd9cf56e62c-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.763734 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.763692 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.763968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.763938 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.764029 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.763998 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.764127 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.764109 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.764216 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.764172 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.764283 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.764227 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.764283 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.764255 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5567046-aa24-421a-8339-fdd9cf56e62c-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.764611 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.764590 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.764764 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.764741 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.764830 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.764793 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.766111 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.766087 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.767139 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.767124 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.787981 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.787958 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.788379 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.788358 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gmmt\" (UniqueName: \"kubernetes.io/projected/d5567046-aa24-421a-8339-fdd9cf56e62c-kube-api-access-4gmmt\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:31.879748 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.879677 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:07:31.879748 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.879724 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" podUID="b664f4ca-736e-4cb1-991d-985d1e6aff34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 18:07:31.953932 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:31.953884 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:32.275803 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:32.275773 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:07:32.275966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:32.275824 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" podUID="58e2469b-dc0d-40ab-bd74-a47c4a9e438b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 18:07:32.319561 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:32.318925 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc"] Oct 30 18:07:32.320493 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:07:32.320461 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5567046_aa24_421a_8339_fdd9cf56e62c.slice/crio-9f278f67587af701c1136ee6b46575434ef728bf9cb94fa8a43e555ff5979637 WatchSource:0}: Error finding container 9f278f67587af701c1136ee6b46575434ef728bf9cb94fa8a43e555ff5979637: Status 404 returned error can't find the container with id 9f278f67587af701c1136ee6b46575434ef728bf9cb94fa8a43e555ff5979637 Oct 30 18:07:32.324387 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:32.324352 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:07:32.324483 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:32.324425 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:07:32.324483 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:32.324460 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:07:32.657490 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:32.657401 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" event={"ID":"d5567046-aa24-421a-8339-fdd9cf56e62c","Type":"ContainerStarted","Data":"ac2d2ba01f24fe73bc3985faf348fec2383e49bb1f192b3800f6546e07025dfc"} Oct 30 18:07:32.657932 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:32.657490 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" event={"ID":"d5567046-aa24-421a-8339-fdd9cf56e62c","Type":"ContainerStarted","Data":"9f278f67587af701c1136ee6b46575434ef728bf9cb94fa8a43e555ff5979637"} Oct 30 18:07:32.717724 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:32.717654 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" podStartSLOduration=1.717638167 podStartE2EDuration="1.717638167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:32.710926393 +0000 UTC m=+913.426496279" watchObservedRunningTime="2026-10-30 18:07:32.717638167 +0000 UTC m=+913.433208040" Oct 30 18:07:32.753264 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:32.753227 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:32.885726 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:32.885695 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:32.887087 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:32.887066 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:32.897529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:32.888524 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:07:32.954728 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:32.954651 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:33.276172 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:33.276135 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:07:33.276313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:33.276212 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" podUID="58e2469b-dc0d-40ab-bd74-a47c4a9e438b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 18:07:33.956458 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:33.956416 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 18:07:33.956885 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:33.956481 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" podUID="d5567046-aa24-421a-8339-fdd9cf56e62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 18:07:34.276298 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:34.276269 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:07:34.276458 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:34.276325 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" podUID="58e2469b-dc0d-40ab-bd74-a47c4a9e438b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 18:07:34.341760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:34.341724 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:34.786918 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:34.786890 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-69644977dd-gvwzm"] Oct 30 18:07:34.955045 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:34.955013 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 18:07:34.955218 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:34.955068 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" podUID="d5567046-aa24-421a-8339-fdd9cf56e62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 18:07:35.276238 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:35.276210 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:07:35.276618 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:35.276262 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" podUID="58e2469b-dc0d-40ab-bd74-a47c4a9e438b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 18:07:35.954878 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:35.954816 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 18:07:35.955051 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:35.954899 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" podUID="d5567046-aa24-421a-8339-fdd9cf56e62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 18:07:36.043250 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:36.043218 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:36.276082 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:36.276052 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:07:36.276243 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:36.276113 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" podUID="58e2469b-dc0d-40ab-bd74-a47c4a9e438b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 18:07:36.954638 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:36.954602 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 18:07:36.955029 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:36.954655 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" podUID="d5567046-aa24-421a-8339-fdd9cf56e62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 18:07:37.280579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:37.280553 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:37.280982 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:37.280956 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:37.281690 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:37.281674 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:07:37.646936 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:37.646853 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:37.954701 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:37.954616 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 18:07:37.954701 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:37.954668 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" podUID="d5567046-aa24-421a-8339-fdd9cf56e62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 18:07:38.019679 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:38.019647 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:07:38.112265 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:38.112129 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:38.956529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:38.955961 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 18:07:38.956529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:38.956021 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" podUID="d5567046-aa24-421a-8339-fdd9cf56e62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 18:07:39.792736 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:39.792699 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:39.793202 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:39.793159 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" podUID="94a706c7-905a-4868-ada3-6d2f576829fa" containerName="payload-processing" containerID="cri-o://421bad578b7d43225d74fac5fe7c8bafeaa2f833ab9072ca22d73086f8327ff5" gracePeriod=25 Oct 30 18:07:40.450702 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:40.450655 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:07:40.505638 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:40.505593 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:07:40.727706 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:40.727601 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:07:40.764529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:40.760704 4603 generic.go:356] "Generic (PLEG): container finished" podID="94a706c7-905a-4868-ada3-6d2f576829fa" containerID="421bad578b7d43225d74fac5fe7c8bafeaa2f833ab9072ca22d73086f8327ff5" exitCode=0 Oct 30 18:07:40.764892 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:40.764754 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" event={"ID":"94a706c7-905a-4868-ada3-6d2f576829fa","Type":"ContainerDied","Data":"421bad578b7d43225d74fac5fe7c8bafeaa2f833ab9072ca22d73086f8327ff5"} Oct 30 18:07:41.149529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.126735 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:07:41.149529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.126807 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" podUID="d5567046-aa24-421a-8339-fdd9cf56e62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:07:41.192532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.185231 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:41.192532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.186657 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:41.214948 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.214918 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:07:41.354162 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.354132 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" Oct 30 18:07:41.493369 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.493276 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a706c7-905a-4868-ada3-6d2f576829fa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/94a706c7-905a-4868-ada3-6d2f576829fa-provider-credentials\") pod \"94a706c7-905a-4868-ada3-6d2f576829fa\" (UID: \"94a706c7-905a-4868-ada3-6d2f576829fa\") " Oct 30 18:07:41.494155 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.494127 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94a706c7-905a-4868-ada3-6d2f576829fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a706c7-905a-4868-ada3-6d2f576829fa-plugins-config-volume\") pod \"94a706c7-905a-4868-ada3-6d2f576829fa\" (UID: \"94a706c7-905a-4868-ada3-6d2f576829fa\") " Oct 30 18:07:41.494360 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.494341 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94a706c7-905a-4868-ada3-6d2f576829fa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94a706c7-905a-4868-ada3-6d2f576829fa-routing-overlay-volume\") pod \"94a706c7-905a-4868-ada3-6d2f576829fa\" (UID: \"94a706c7-905a-4868-ada3-6d2f576829fa\") " Oct 30 18:07:41.494531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.494494 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a706c7-905a-4868-ada3-6d2f576829fa-kube-api-access-qh7xf\" (UniqueName: \"kubernetes.io/projected/94a706c7-905a-4868-ada3-6d2f576829fa-kube-api-access-qh7xf\") pod \"94a706c7-905a-4868-ada3-6d2f576829fa\" (UID: \"94a706c7-905a-4868-ada3-6d2f576829fa\") " Oct 30 18:07:41.521933 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.521885 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a706c7-905a-4868-ada3-6d2f576829fa-routing-overlay-volume" pod "94a706c7-905a-4868-ada3-6d2f576829fa" (UID: "94a706c7-905a-4868-ada3-6d2f576829fa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:07:41.530112 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.530060 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a706c7-905a-4868-ada3-6d2f576829fa-plugins-config-volume" pod "94a706c7-905a-4868-ada3-6d2f576829fa" (UID: "94a706c7-905a-4868-ada3-6d2f576829fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:07:41.536025 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.535998 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a706c7-905a-4868-ada3-6d2f576829fa-kube-api-access-qh7xf" pod "94a706c7-905a-4868-ada3-6d2f576829fa" (UID: "94a706c7-905a-4868-ada3-6d2f576829fa"). InnerVolumeSpecName "kube-api-access-qh7xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:41.556420 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.556385 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a706c7-905a-4868-ada3-6d2f576829fa-provider-credentials" pod "94a706c7-905a-4868-ada3-6d2f576829fa" (UID: "94a706c7-905a-4868-ada3-6d2f576829fa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:41.580947 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.580913 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:41.729530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.647406 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:07:41.729530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.674718 4603 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94a706c7-905a-4868-ada3-6d2f576829fa-provider-credentials\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:07:41.729530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.715896 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a706c7-905a-4868-ada3-6d2f576829fa-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:07:41.729530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.715923 4603 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94a706c7-905a-4868-ada3-6d2f576829fa-routing-overlay-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:07:41.729530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.715951 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh7xf\" (UniqueName: \"kubernetes.io/projected/94a706c7-905a-4868-ada3-6d2f576829fa-kube-api-access-qh7xf\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:07:41.729530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.716027 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:07:41.756685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.741132 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:07:41.756685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.741943 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:07:41.802357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.779934 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:07:41.898193 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.898149 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:07:41.940257 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:41.940189 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:07:42.146464 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:42.146433 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" Oct 30 18:07:42.174569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:42.163248 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69644977dd-gvwzm" event={"ID":"94a706c7-905a-4868-ada3-6d2f576829fa","Type":"ContainerDied","Data":"e62ab2557f7c37aedbdb0b18e9b7dbb3f7be3e29d4454506fb404573dbf9c299"} Oct 30 18:07:42.174894 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:42.174870 4603 scope.go:122] "RemoveContainer" containerID="421bad578b7d43225d74fac5fe7c8bafeaa2f833ab9072ca22d73086f8327ff5" Oct 30 18:07:42.290251 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:42.290186 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-69644977dd-gvwzm"] Oct 30 18:07:42.333423 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:42.333391 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-69644977dd-gvwzm"] Oct 30 18:07:43.414608 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:43.414568 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:43.516525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:43.510893 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94a706c7-905a-4868-ada3-6d2f576829fa" path="/var/lib/kubelet/pods/94a706c7-905a-4868-ada3-6d2f576829fa/volumes" Oct 30 18:07:44.947143 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:44.947112 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:46.633663 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:46.633631 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:48.369085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:48.366193 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:50.131767 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:50.131733 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:51.799739 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:51.799707 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:51.934670 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:07:51.934627 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:07:53.445290 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:53.445255 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:55.048335 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:55.048301 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:55.783064 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:55.783035 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf"] Oct 30 18:07:55.783666 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:55.783642 4603 cpu_manager.go:414] "containerMap: removing container" podUID="94a706c7-905a-4868-ada3-6d2f576829fa" containerName="payload-processing" Oct 30 18:07:55.783666 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:55.783667 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a706c7-905a-4868-ada3-6d2f576829fa" containerName="payload-processing" Oct 30 18:07:55.783844 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:55.783793 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a706c7-905a-4868-ada3-6d2f576829fa" containerName="payload-processing" Oct 30 18:07:55.788770 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:55.788744 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:07:55.806872 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:55.806849 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf"] Oct 30 18:07:55.815356 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:55.815336 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b424a5\"/\"distinct-w1-b424a5-kserve-self-signed-certs\"" Oct 30 18:07:55.878728 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:55.878704 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-tls-certs\") pod \"distinct-w1-b424a5-kserve-78c94b7886-6kqxf\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:07:55.878866 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:55.878743 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lccs\" (UniqueName: \"kubernetes.io/projected/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-kube-api-access-2lccs\") pod \"distinct-w1-b424a5-kserve-78c94b7886-6kqxf\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:07:55.878866 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:55.878850 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-home\") pod \"distinct-w1-b424a5-kserve-78c94b7886-6kqxf\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:07:55.878996 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:55.878981 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-dshm\") pod \"distinct-w1-b424a5-kserve-78c94b7886-6kqxf\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:07:55.879056 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:55.879023 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-model-cache\") pod \"distinct-w1-b424a5-kserve-78c94b7886-6kqxf\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:07:55.879214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:55.879189 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-tmp-dir\") pod \"distinct-w1-b424a5-kserve-78c94b7886-6kqxf\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:07:55.882534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:55.882495 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:55.996528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:55.992317 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-dshm\") pod \"distinct-w1-b424a5-kserve-78c94b7886-6kqxf\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:07:55.996528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:55.992375 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-model-cache\") pod \"distinct-w1-b424a5-kserve-78c94b7886-6kqxf\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:07:55.996528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:55.992995 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-model-cache\") pod \"distinct-w1-b424a5-kserve-78c94b7886-6kqxf\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:07:55.996528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:55.993436 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-tmp-dir\") pod \"distinct-w1-b424a5-kserve-78c94b7886-6kqxf\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:07:55.996528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:55.993528 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-tls-certs\") pod \"distinct-w1-b424a5-kserve-78c94b7886-6kqxf\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:07:55.996528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:55.993560 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lccs\" (UniqueName: \"kubernetes.io/projected/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-kube-api-access-2lccs\") pod \"distinct-w1-b424a5-kserve-78c94b7886-6kqxf\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:07:55.996528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:55.993613 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-home\") pod \"distinct-w1-b424a5-kserve-78c94b7886-6kqxf\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:07:56.005526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:56.000916 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-tmp-dir\") pod \"distinct-w1-b424a5-kserve-78c94b7886-6kqxf\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:07:56.005526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:56.001704 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-home\") pod \"distinct-w1-b424a5-kserve-78c94b7886-6kqxf\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:07:56.038066 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:56.037981 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-dshm\") pod \"distinct-w1-b424a5-kserve-78c94b7886-6kqxf\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:07:56.043796 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:56.043753 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lccs\" (UniqueName: \"kubernetes.io/projected/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-kube-api-access-2lccs\") pod \"distinct-w1-b424a5-kserve-78c94b7886-6kqxf\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:07:56.055172 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:56.055141 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-tls-certs\") pod \"distinct-w1-b424a5-kserve-78c94b7886-6kqxf\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:07:56.114526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:56.114018 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:07:56.434016 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:56.433989 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf"] Oct 30 18:07:56.435472 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:07:56.435446 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3e90a36_bf75_4bfa_9f0c_d75dc709f4d9.slice/crio-17d926044da064d3f3edbec0dd54efaa09c1554cd0e03167bfa03df1815fc7ca WatchSource:0}: Error finding container 17d926044da064d3f3edbec0dd54efaa09c1554cd0e03167bfa03df1815fc7ca: Status 404 returned error can't find the container with id 17d926044da064d3f3edbec0dd54efaa09c1554cd0e03167bfa03df1815fc7ca Oct 30 18:07:56.827543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:56.827483 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:57.274909 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:57.274872 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" event={"ID":"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9","Type":"ContainerStarted","Data":"5cb353540add350c5f2dbf1340485e8b8652709eb34c434a65ed2f2a1fb277ba"} Oct 30 18:07:57.275358 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:57.274925 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" event={"ID":"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9","Type":"ContainerStarted","Data":"17d926044da064d3f3edbec0dd54efaa09c1554cd0e03167bfa03df1815fc7ca"} Oct 30 18:07:57.317402 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:57.317352 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podStartSLOduration=2.317337578 podStartE2EDuration="2.317337578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:57.314423613 +0000 UTC m=+938.029993486" watchObservedRunningTime="2026-10-30 18:07:57.317337578 +0000 UTC m=+938.032907452" Oct 30 18:07:58.114575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:58.114547 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:07:58.537819 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:07:58.537788 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:00.242566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:00.242534 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:01.943919 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:01.943883 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:03.456669 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:03.456635 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:03.670595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:03.668604 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:04.606448 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.606416 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr"] Oct 30 18:08:04.611465 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.611446 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.637914 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.637889 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr"] Oct 30 18:08:04.707926 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.707890 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:04.759032 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.759008 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.759200 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.759042 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.759200 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.759069 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.759334 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.759310 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d6cd466-9f15-476d-9f04-6d03176f025e-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.759399 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.759362 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz8kq\" (UniqueName: \"kubernetes.io/projected/4d6cd466-9f15-476d-9f04-6d03176f025e-kube-api-access-hz8kq\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.759465 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.759414 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.759536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.759489 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.759601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.759561 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.759655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.759596 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.860392 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.860321 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d6cd466-9f15-476d-9f04-6d03176f025e-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.860392 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.860354 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz8kq\" (UniqueName: \"kubernetes.io/projected/4d6cd466-9f15-476d-9f04-6d03176f025e-kube-api-access-hz8kq\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.860392 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.860387 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.860664 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.860414 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.860664 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.860456 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.860664 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.860488 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.860664 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.860611 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.860664 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.860653 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.860950 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.860698 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.861023 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.860991 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.861084 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.861032 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.861187 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.861135 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.861187 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.861135 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d6cd466-9f15-476d-9f04-6d03176f025e-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.861187 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.861178 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.862949 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.862923 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.863136 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.863121 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.874291 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.874238 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:04.883019 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.882996 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.883110 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.883071 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz8kq\" (UniqueName: \"kubernetes.io/projected/4d6cd466-9f15-476d-9f04-6d03176f025e-kube-api-access-hz8kq\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:04.926003 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:04.925984 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:05.409526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:05.390636 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr"] Oct 30 18:08:05.455628 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:05.454056 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:05.455628 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:05.454222 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:05.455628 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:05.454281 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:05.667661 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:05.667568 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:06.114903 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:06.114876 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:08:06.127238 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:06.127210 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:08:06.458898 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:06.458815 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" event={"ID":"4d6cd466-9f15-476d-9f04-6d03176f025e","Type":"ContainerStarted","Data":"eb27aae561bd07172ee2db263bba09e608090627f817261fe086fcbe24d81326"} Oct 30 18:08:06.458898 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:06.458872 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" event={"ID":"4d6cd466-9f15-476d-9f04-6d03176f025e","Type":"ContainerStarted","Data":"1da322e0f9cdb7814bf0a103260960be17f1a8e6bdaa5196b757c6308e8042fb"} Oct 30 18:08:06.470292 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:06.470264 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:08:06.517063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:06.517012 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" podStartSLOduration=2.516996369 podStartE2EDuration="2.516996369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:06.510927642 +0000 UTC m=+947.226497527" watchObservedRunningTime="2026-10-30 18:08:06.516996369 +0000 UTC m=+947.232566243" Oct 30 18:08:06.926944 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:06.926913 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:06.927966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:06.927946 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:08:06.928030 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:06.927992 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" podUID="4d6cd466-9f15-476d-9f04-6d03176f025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:08:07.478237 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:07.478204 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:07.926926 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:07.926895 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:08:07.927120 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:07.926955 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" podUID="4d6cd466-9f15-476d-9f04-6d03176f025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:08:08.116366 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:08.115932 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:08.880731 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:08.880698 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:08:08.927338 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:08.927307 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:08:08.927773 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:08.927360 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" podUID="4d6cd466-9f15-476d-9f04-6d03176f025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:08:09.212299 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:09.212206 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:09.927269 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:09.927226 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:08:09.927493 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:09.927276 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" podUID="4d6cd466-9f15-476d-9f04-6d03176f025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:08:10.941626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:10.941587 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:08:10.942226 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:10.942193 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" podUID="4d6cd466-9f15-476d-9f04-6d03176f025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:08:11.354067 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:11.354032 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:11.930910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:11.930877 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:08:11.932134 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:11.932105 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" podUID="4d6cd466-9f15-476d-9f04-6d03176f025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:08:12.310948 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:12.303559 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:12.394557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:12.392705 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:12.471541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:12.464647 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:12.906689 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:12.906633 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:08:12.968530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:12.960634 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:13.041530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:12.995941 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:08:13.067657 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:13.067607 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" podUID="4d6cd466-9f15-476d-9f04-6d03176f025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:08:13.367541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:13.360779 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:08:13.408266 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:13.408225 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:13.410468 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:13.410446 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:08:13.464941 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:13.464901 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:13.495531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:13.485430 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:08:13.495531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:13.485488 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:08:13.729513 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:13.729405 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:13.774656 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:13.773544 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:13.910530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:13.908609 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:13.910530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:13.909945 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:08:13.938701 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:13.938036 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:08:13.938701 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:13.938098 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" podUID="4d6cd466-9f15-476d-9f04-6d03176f025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:08:14.094391 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:14.094319 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:08:14.094756 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:14.094698 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" podUID="22e0b280-2b65-4fad-8344-67049190d108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:14.388724 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:14.388621 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:14.410027 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:14.409983 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:14.473655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:14.473590 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:14.784017 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:14.774889 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:14.784017 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:14.775247 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:08:14.784017 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:14.775705 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:08:14.954687 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:14.954636 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:14.963530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:14.955958 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:08:14.988551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:14.956592 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:08:14.988551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:14.983710 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" podUID="81f49829-73ed-4760-9444-f1793861eba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:14.988551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:14.975767 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:08:14.988551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:14.984776 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" podUID="4d6cd466-9f15-476d-9f04-6d03176f025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:08:15.062136 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:15.062031 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:15.150264 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:15.150228 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:08:15.150585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:15.150552 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" podUID="571fc9dd-71e3-4fa7-b1af-e74f0a96654d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:15.383350 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:15.383260 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:08:15.532553 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:15.532469 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:08:15.558184 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:15.557880 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:08:15.610129 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:15.597753 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:08:15.637609 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:15.635714 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:08:15.912586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:15.912470 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:15.938324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:15.938289 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:08:15.942647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:15.942602 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" podUID="4d6cd466-9f15-476d-9f04-6d03176f025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:08:16.191381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:16.191287 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:08:16.353560 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:16.352881 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:08:16.354672 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:16.354647 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:16.542460 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:16.542428 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:08:17.470752 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:17.470721 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:17.955112 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:17.955077 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:08:17.955652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:17.955141 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" podUID="4d6cd466-9f15-476d-9f04-6d03176f025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:17.959406 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:17.959381 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:17.960535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:17.960273 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:17.961322 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:17.961302 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:08:19.267639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:19.267600 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:20.914332 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:20.914299 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:21.931682 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:08:21.931645 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:08:22.609736 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:22.609704 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:24.271568 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.271490 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:24.764617 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.764544 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v"] Oct 30 18:08:24.769913 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.769895 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.792840 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.792803 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v"] Oct 30 18:08:24.870785 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.870758 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.870933 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.870802 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.870933 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.870824 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.870933 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.870889 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktqqm\" (UniqueName: \"kubernetes.io/projected/d4a3adf6-fb23-4334-8bab-f800b4e8e089-kube-api-access-ktqqm\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.871072 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.870990 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.871108 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.871080 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.871108 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.871102 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.871213 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.871194 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.871253 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.871222 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.973004 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.972973 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.973153 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.973067 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.973153 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.973093 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.973244 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.973170 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.973244 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.973196 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.973332 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.973279 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.973332 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.973327 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.973432 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.973361 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.973432 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.973364 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.973432 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.973388 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktqqm\" (UniqueName: \"kubernetes.io/projected/d4a3adf6-fb23-4334-8bab-f800b4e8e089-kube-api-access-ktqqm\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.973605 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.973455 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.973798 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.973776 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.973929 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.973906 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.974009 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.973981 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.976185 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.976162 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.976304 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.976287 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.996445 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.996425 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:24.996567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:24.996496 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktqqm\" (UniqueName: \"kubernetes.io/projected/d4a3adf6-fb23-4334-8bab-f800b4e8e089-kube-api-access-ktqqm\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:25.086162 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:25.086143 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:25.409853 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:25.409826 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v"] Oct 30 18:08:25.410349 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:08:25.410324 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a3adf6_fb23_4334_8bab_f800b4e8e089.slice/crio-f6ae81debcf20b88835927277b634002f56de72cc703590641a84677bcab4bad WatchSource:0}: Error finding container f6ae81debcf20b88835927277b634002f56de72cc703590641a84677bcab4bad: Status 404 returned error can't find the container with id f6ae81debcf20b88835927277b634002f56de72cc703590641a84677bcab4bad Oct 30 18:08:25.412570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:25.412541 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:25.412654 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:25.412601 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:25.412654 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:25.412630 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:25.933274 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:25.933239 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:26.287731 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:26.287696 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" event={"ID":"d4a3adf6-fb23-4334-8bab-f800b4e8e089","Type":"ContainerStarted","Data":"3fb29526621b83ed763d3401b3b37a008321bb644366f41e7cbc405cadeef19c"} Oct 30 18:08:26.287895 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:26.287779 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" event={"ID":"d4a3adf6-fb23-4334-8bab-f800b4e8e089","Type":"ContainerStarted","Data":"f6ae81debcf20b88835927277b634002f56de72cc703590641a84677bcab4bad"} Oct 30 18:08:26.345835 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:26.345783 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" podStartSLOduration=2.345769248 podStartE2EDuration="2.345769248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:26.344139866 +0000 UTC m=+967.059709745" watchObservedRunningTime="2026-10-30 18:08:26.345769248 +0000 UTC m=+967.061339189" Oct 30 18:08:27.087871 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:27.087837 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:27.100524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:27.089822 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:08:27.100524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:27.089888 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" podUID="d4a3adf6-fb23-4334-8bab-f800b4e8e089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 18:08:27.876535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:27.874147 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:28.086953 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:28.086921 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:08:28.087115 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:28.086975 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" podUID="d4a3adf6-fb23-4334-8bab-f800b4e8e089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 18:08:29.086630 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:29.086596 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:08:29.087015 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:29.086655 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" podUID="d4a3adf6-fb23-4334-8bab-f800b4e8e089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 18:08:29.840540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:29.838536 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:30.086437 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:30.086400 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:08:30.086646 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:30.086457 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" podUID="d4a3adf6-fb23-4334-8bab-f800b4e8e089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 18:08:31.087407 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:31.087377 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:08:31.087831 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:31.087439 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" podUID="d4a3adf6-fb23-4334-8bab-f800b4e8e089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 18:08:31.720708 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:31.720677 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:32.086810 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:32.086776 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:08:32.086964 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:32.086843 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" podUID="d4a3adf6-fb23-4334-8bab-f800b4e8e089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 18:08:33.126063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:33.126033 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:33.126811 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:33.126688 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:33.128125 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:33.128095 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:08:33.712825 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:33.712768 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:34.173857 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.173819 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2"] Oct 30 18:08:34.180233 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.180209 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:34.196858 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.196831 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2"] Oct 30 18:08:34.204773 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.204753 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3cc458\"/\"distinct-w3-3cc458-kserve-self-signed-certs\"" Oct 30 18:08:34.227402 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.227376 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/521cc860-6923-45a7-8823-4859624c9267-tls-certs\") pod \"distinct-w3-3cc458-kserve-c9d86cb78-ps8j2\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:34.227526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.227408 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-tmp-dir\") pod \"distinct-w3-3cc458-kserve-c9d86cb78-ps8j2\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:34.227620 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.227599 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb89j\" (UniqueName: \"kubernetes.io/projected/521cc860-6923-45a7-8823-4859624c9267-kube-api-access-gb89j\") pod \"distinct-w3-3cc458-kserve-c9d86cb78-ps8j2\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:34.227717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.227702 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-home\") pod \"distinct-w3-3cc458-kserve-c9d86cb78-ps8j2\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:34.227757 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.227747 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-dshm\") pod \"distinct-w3-3cc458-kserve-c9d86cb78-ps8j2\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:34.227854 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.227827 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-model-cache\") pod \"distinct-w3-3cc458-kserve-c9d86cb78-ps8j2\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:34.301943 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.301906 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:34.328530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.328480 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-home\") pod \"distinct-w3-3cc458-kserve-c9d86cb78-ps8j2\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:34.328620 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.328572 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-dshm\") pod \"distinct-w3-3cc458-kserve-c9d86cb78-ps8j2\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:34.328661 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.328621 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-model-cache\") pod \"distinct-w3-3cc458-kserve-c9d86cb78-ps8j2\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:34.328708 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.328693 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/521cc860-6923-45a7-8823-4859624c9267-tls-certs\") pod \"distinct-w3-3cc458-kserve-c9d86cb78-ps8j2\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:34.328755 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.328728 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-tmp-dir\") pod \"distinct-w3-3cc458-kserve-c9d86cb78-ps8j2\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:34.328828 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.328797 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-home\") pod \"distinct-w3-3cc458-kserve-c9d86cb78-ps8j2\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:34.328828 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.328824 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb89j\" (UniqueName: \"kubernetes.io/projected/521cc860-6923-45a7-8823-4859624c9267-kube-api-access-gb89j\") pod \"distinct-w3-3cc458-kserve-c9d86cb78-ps8j2\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:34.329028 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.329004 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-model-cache\") pod \"distinct-w3-3cc458-kserve-c9d86cb78-ps8j2\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:34.329082 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.329069 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-tmp-dir\") pod \"distinct-w3-3cc458-kserve-c9d86cb78-ps8j2\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:34.331038 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.331020 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-dshm\") pod \"distinct-w3-3cc458-kserve-c9d86cb78-ps8j2\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:34.331220 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.331203 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/521cc860-6923-45a7-8823-4859624c9267-tls-certs\") pod \"distinct-w3-3cc458-kserve-c9d86cb78-ps8j2\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:34.355310 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.355289 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb89j\" (UniqueName: \"kubernetes.io/projected/521cc860-6923-45a7-8823-4859624c9267-kube-api-access-gb89j\") pod \"distinct-w3-3cc458-kserve-c9d86cb78-ps8j2\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:34.498351 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.497953 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:34.890233 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:34.890205 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2"] Oct 30 18:08:34.891517 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:08:34.891469 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod521cc860_6923_45a7_8823_4859624c9267.slice/crio-5d90a2d95edd73ca43b50d846fa6c22e01ac043f6ce0e7f52a5f42930a8ad50f WatchSource:0}: Error finding container 5d90a2d95edd73ca43b50d846fa6c22e01ac043f6ce0e7f52a5f42930a8ad50f: Status 404 returned error can't find the container with id 5d90a2d95edd73ca43b50d846fa6c22e01ac043f6ce0e7f52a5f42930a8ad50f Oct 30 18:08:35.364544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:35.364475 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" event={"ID":"521cc860-6923-45a7-8823-4859624c9267","Type":"ContainerStarted","Data":"74b6b90807e273cd6f183f0ee624b7d5461d694fd7caaa37c9109a3d645dd181"} Oct 30 18:08:35.364901 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:35.364586 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" event={"ID":"521cc860-6923-45a7-8823-4859624c9267","Type":"ContainerStarted","Data":"5d90a2d95edd73ca43b50d846fa6c22e01ac043f6ce0e7f52a5f42930a8ad50f"} Oct 30 18:08:35.406452 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:35.406409 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podStartSLOduration=1.406396214 podStartE2EDuration="1.406396214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:35.404752234 +0000 UTC m=+976.120322109" watchObservedRunningTime="2026-10-30 18:08:35.406396214 +0000 UTC m=+976.121966088" Oct 30 18:08:35.499664 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:35.499635 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:35.837934 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:35.837896 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:37.098358 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.098319 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc"] Oct 30 18:08:37.103068 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.103047 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:37.124089 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.124062 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc"] Oct 30 18:08:37.131715 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.131693 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b424a5\"/\"distinct-2-w1-b424a5-kserve-self-signed-certs\"" Oct 30 18:08:37.188427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.188397 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:37.195181 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.195155 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnzp6\" (UniqueName: \"kubernetes.io/projected/bb34381d-3c03-4aa1-9af1-7d000b687af0-kube-api-access-nnzp6\") pod \"distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:37.195280 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.195200 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb34381d-3c03-4aa1-9af1-7d000b687af0-tls-certs\") pod \"distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:37.195353 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.195333 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-home\") pod \"distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:37.195406 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.195389 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-model-cache\") pod \"distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:37.195482 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.195469 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-tmp-dir\") pod \"distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:37.195534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.195511 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-dshm\") pod \"distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:37.311556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.305446 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-home\") pod \"distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:37.311556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.305558 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-model-cache\") pod \"distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:37.311556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.305615 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-tmp-dir\") pod \"distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:37.311556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.305643 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-dshm\") pod \"distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:37.311556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.305848 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnzp6\" (UniqueName: \"kubernetes.io/projected/bb34381d-3c03-4aa1-9af1-7d000b687af0-kube-api-access-nnzp6\") pod \"distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:37.311556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.305898 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb34381d-3c03-4aa1-9af1-7d000b687af0-tls-certs\") pod \"distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:37.311556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.307231 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-tmp-dir\") pod \"distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:37.311556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.307519 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-home\") pod \"distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:37.311556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.307763 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-model-cache\") pod \"distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:37.363035 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.350312 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnzp6\" (UniqueName: \"kubernetes.io/projected/bb34381d-3c03-4aa1-9af1-7d000b687af0-kube-api-access-nnzp6\") pod \"distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:37.363301 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.356103 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb34381d-3c03-4aa1-9af1-7d000b687af0-tls-certs\") pod \"distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:37.366277 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.365829 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-dshm\") pod \"distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:37.422528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.417878 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:37.783107 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.783086 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc"] Oct 30 18:08:37.784342 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:08:37.784319 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb34381d_3c03_4aa1_9af1_7d000b687af0.slice/crio-bfc2a551d2fad09e8441f6393bb28f8b0014fc9f07f471a537980964960196c6 WatchSource:0}: Error finding container bfc2a551d2fad09e8441f6393bb28f8b0014fc9f07f471a537980964960196c6: Status 404 returned error can't find the container with id bfc2a551d2fad09e8441f6393bb28f8b0014fc9f07f471a537980964960196c6 Oct 30 18:08:37.822567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:37.822540 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:38.402905 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:38.402874 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" event={"ID":"bb34381d-3c03-4aa1-9af1-7d000b687af0","Type":"ContainerStarted","Data":"c588e962ba8430936ef1325b3f47796b8318e25ee6fa15594678d697d6bedfe1"} Oct 30 18:08:38.402905 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:38.402910 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" event={"ID":"bb34381d-3c03-4aa1-9af1-7d000b687af0","Type":"ContainerStarted","Data":"bfc2a551d2fad09e8441f6393bb28f8b0014fc9f07f471a537980964960196c6"} Oct 30 18:08:38.424328 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:38.424304 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:38.445357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:38.445309 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podStartSLOduration=1.445296913 podStartE2EDuration="1.445296913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:38.443325214 +0000 UTC m=+979.158895088" watchObservedRunningTime="2026-10-30 18:08:38.445296913 +0000 UTC m=+979.160866786" Oct 30 18:08:39.601532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:39.600496 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:39.813052 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:39.813022 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x"] Oct 30 18:08:39.816950 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:39.816930 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:39.847376 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:39.847352 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x"] Oct 30 18:08:39.962182 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:39.962100 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:39.962182 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:39.962158 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:39.962357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:39.962216 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:39.962357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:39.962241 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:39.962357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:39.962348 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:39.962455 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:39.962383 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:39.962571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:39.962448 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvrsm\" (UniqueName: \"kubernetes.io/projected/1d187b51-f5c6-4a63-8300-2f01184a15d3-kube-api-access-xvrsm\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:39.962571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:39.962528 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:39.962643 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:39.962580 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d187b51-f5c6-4a63-8300-2f01184a15d3-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:40.063485 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:40.063459 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:40.063667 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:40.063551 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d187b51-f5c6-4a63-8300-2f01184a15d3-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:40.063667 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:40.063619 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:40.063667 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:40.063658 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:40.063834 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:40.063673 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:40.063834 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:40.063704 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:40.063834 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:40.063791 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:40.063986 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:40.063832 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:40.063986 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:40.063884 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvrsm\" (UniqueName: \"kubernetes.io/projected/1d187b51-f5c6-4a63-8300-2f01184a15d3-kube-api-access-xvrsm\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:40.064091 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:40.064034 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:40.064148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:40.064120 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:40.064294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:40.064276 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d187b51-f5c6-4a63-8300-2f01184a15d3-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:40.064349 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:40.064327 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:40.064409 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:40.064395 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:40.066274 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:40.066254 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:40.066568 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:40.066548 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:40.086303 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:40.086280 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvrsm\" (UniqueName: \"kubernetes.io/projected/1d187b51-f5c6-4a63-8300-2f01184a15d3-kube-api-access-xvrsm\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:40.086389 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:40.086350 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:40.133359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:40.133337 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:40.458120 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:40.458085 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x"] Oct 30 18:08:40.459425 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:08:40.459398 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d187b51_f5c6_4a63_8300_2f01184a15d3.slice/crio-c4980b95d79b0465d22191c7fc381400920ea6f1aca149eb74ec1b1a67c535e2 WatchSource:0}: Error finding container c4980b95d79b0465d22191c7fc381400920ea6f1aca149eb74ec1b1a67c535e2: Status 404 returned error can't find the container with id c4980b95d79b0465d22191c7fc381400920ea6f1aca149eb74ec1b1a67c535e2 Oct 30 18:08:40.461861 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:40.461826 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:40.461959 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:40.461902 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:40.461959 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:40.461947 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:41.251668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:41.251635 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:41.438303 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:41.438274 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" event={"ID":"1d187b51-f5c6-4a63-8300-2f01184a15d3","Type":"ContainerStarted","Data":"f65aac4dd93bea107473a1364b0f53516964b3092ac1cdbb6cfbf7db84119f14"} Oct 30 18:08:41.438472 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:41.438323 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" event={"ID":"1d187b51-f5c6-4a63-8300-2f01184a15d3","Type":"ContainerStarted","Data":"c4980b95d79b0465d22191c7fc381400920ea6f1aca149eb74ec1b1a67c535e2"} Oct 30 18:08:41.522618 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:41.522560 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" podStartSLOduration=2.522544224 podStartE2EDuration="2.522544224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:41.508967525 +0000 UTC m=+982.224537424" watchObservedRunningTime="2026-10-30 18:08:41.522544224 +0000 UTC m=+982.238114097" Oct 30 18:08:42.133784 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:42.133752 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:42.135512 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:42.135479 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 18:08:42.135608 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:42.135538 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" podUID="1d187b51-f5c6-4a63-8300-2f01184a15d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 18:08:42.937272 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:42.937237 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:43.133920 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:43.133890 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 18:08:43.134133 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:43.133951 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" podUID="1d187b51-f5c6-4a63-8300-2f01184a15d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 18:08:44.134458 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:44.134425 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 18:08:44.134883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:44.134491 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" podUID="1d187b51-f5c6-4a63-8300-2f01184a15d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 18:08:44.509580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:44.508908 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:44.526834 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:44.526807 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:44.535954 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:44.535925 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:45.134774 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:45.134736 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 18:08:45.135178 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:45.134816 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" podUID="1d187b51-f5c6-4a63-8300-2f01184a15d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 18:08:45.471831 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:45.471758 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:46.134260 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:46.134223 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 18:08:46.134438 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:46.134287 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" podUID="1d187b51-f5c6-4a63-8300-2f01184a15d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 18:08:46.342794 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:46.342760 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:47.133969 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:47.133937 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 18:08:47.134120 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:47.134013 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" podUID="1d187b51-f5c6-4a63-8300-2f01184a15d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 18:08:47.424867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:47.424778 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:47.437587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:47.437563 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:47.487201 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:47.487168 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:47.936573 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:47.936538 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:48.137438 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:48.137413 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:48.137912 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:48.137894 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:48.138548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:48.138532 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:08:49.611612 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:49.611582 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:51.381528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:51.381031 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:51.930206 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:08:51.930171 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:08:52.939656 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:52.939621 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:53.663249 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:53.663219 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:08:54.584849 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:54.584813 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:56.384706 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:56.384651 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:56.397647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:56.397606 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:56.465011 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:56.464934 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:56.477603 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:56.477569 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:08:56.560629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:56.560585 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:56.955702 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:56.955648 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:08:57.010638 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:57.010588 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:57.185691 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:57.185638 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:57.344648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:57.344592 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:57.378709 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:57.378229 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:57.413764 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:57.413720 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:08:57.439007 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:57.438979 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:08:57.536161 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:57.536119 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:58.047579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:58.036787 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:08:58.449072 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:58.447609 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:58.486307 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:58.484757 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:58.499531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:58.486578 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 30 18:08:58.540627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:58.540578 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:58.567756 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:58.567712 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:58.662305 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:58.661608 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:08:58.678738 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:58.678614 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:58.957650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:58.957602 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:58.981976 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:58.981903 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:59.450734 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:59.450691 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:59.488527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:59.485062 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:59.488527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:59.486346 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:08:59.614538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:59.572430 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:59.737609 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:59.629758 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:59.737609 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:59.629793 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:59.737609 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:59.675908 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:59.737609 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:59.715046 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:08:59.737609 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:59.719992 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:08:59.867655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:59.867616 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:08:59.867937 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:59.867906 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" podUID="81f49829-73ed-4760-9444-f1793861eba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:59.999594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:59.991975 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:00.058537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:00.025559 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:09:00.058537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:08:59.992575 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:00.058537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:00.026978 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:09:00.309585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:00.288795 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Oct 30 18:09:00.476824 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:00.476785 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:00.496222 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:00.495568 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:00.696495 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:00.693994 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:09:00.720527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:00.717185 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:09:00.720527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:00.717811 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:09:00.724525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:00.723520 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:09:00.792566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:00.767800 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:01.040528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:01.032798 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:09:01.040528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:01.035916 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:01.040528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:01.036083 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:09:01.534580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:01.519223 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:09:01.577198 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:01.577156 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:01.699530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:01.697668 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:01.770738 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:01.770694 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:01.783970 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:01.783928 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:01.839576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:01.786304 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:09:01.924985 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:01.924958 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:09:02.039863 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:02.039824 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:09:02.041386 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:02.041364 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:09:02.041747 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:02.041728 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:09:02.041910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:02.041896 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:09:02.042058 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:02.042044 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:09:02.095591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:02.094554 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:09:02.784888 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:02.784833 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:04.419373 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:04.419335 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:06.052768 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:06.052726 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:06.525420 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:06.525386 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:07.742772 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:07.742738 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:09.476138 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.476097 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:09.684908 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.684879 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk"] Oct 30 18:09:09.688925 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.688904 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.717520 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.717474 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk"] Oct 30 18:09:09.733208 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.733128 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.733208 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.733174 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.733208 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.733204 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.733434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.733229 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jl4z\" (UniqueName: \"kubernetes.io/projected/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-kube-api-access-9jl4z\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.733434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.733263 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.733434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.733348 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.733596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.733456 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.733596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.733577 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.733666 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.733624 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.834788 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.834754 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.834966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.834857 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.834966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.834898 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.834966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.834956 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.835138 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.834983 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.835138 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.835013 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.835138 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.835039 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jl4z\" (UniqueName: \"kubernetes.io/projected/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-kube-api-access-9jl4z\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.835138 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.835077 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.835346 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.835255 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.835346 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.835305 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.835565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.835529 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.835652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.835616 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.835778 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.835671 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.835778 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.835714 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.837410 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.837390 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.837566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.837549 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.867435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.867401 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:09.867760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:09.867743 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jl4z\" (UniqueName: \"kubernetes.io/projected/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-kube-api-access-9jl4z\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:10.003541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:10.003466 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:10.333232 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:10.333208 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk"] Oct 30 18:09:10.334287 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:09:10.334257 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod187c6c98_bcb4_4e66_b394_f5ce4e4c5d9e.slice/crio-1344958c6607e3761f5e253a7cf4f0a9836ca2a84f8da57cb7d73c3fa81be914 WatchSource:0}: Error finding container 1344958c6607e3761f5e253a7cf4f0a9836ca2a84f8da57cb7d73c3fa81be914: Status 404 returned error can't find the container with id 1344958c6607e3761f5e253a7cf4f0a9836ca2a84f8da57cb7d73c3fa81be914 Oct 30 18:09:10.336810 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:10.336776 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:10.336905 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:10.336840 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:10.336905 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:10.336888 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:10.920911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:10.920869 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" event={"ID":"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e","Type":"ContainerStarted","Data":"268d83b09abb47512048569aaa77e68400eea8df484d57fcd9dbd35e1fa8aed9"} Oct 30 18:09:10.921310 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:10.920952 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" event={"ID":"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e","Type":"ContainerStarted","Data":"1344958c6607e3761f5e253a7cf4f0a9836ca2a84f8da57cb7d73c3fa81be914"} Oct 30 18:09:10.983797 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:10.983748 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" podStartSLOduration=1.9837344300000002 podStartE2EDuration="1.98373443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:10.975899695 +0000 UTC m=+1011.691469569" watchObservedRunningTime="2026-10-30 18:09:10.98373443 +0000 UTC m=+1011.699304304" Oct 30 18:09:11.010529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:11.009843 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:11.014662 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:11.014624 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:09:11.015079 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:11.015050 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" podUID="187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 18:09:11.097888 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:11.097853 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:12.006858 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:12.006825 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:09:12.007453 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:12.007423 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" podUID="187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 18:09:12.772372 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:12.772333 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:13.003934 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:13.003894 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:09:13.004220 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:13.004187 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" podUID="187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 18:09:13.734480 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:13.734442 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:14.004923 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:14.004844 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:09:14.005155 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:14.005127 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" podUID="187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 18:09:14.624531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:14.623536 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:15.007965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:15.007874 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:09:15.008589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:15.008554 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" podUID="187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 18:09:16.010527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:16.009831 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:09:16.010527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:16.009896 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" podUID="187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 18:09:16.372185 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:16.372154 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:16.521571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:16.517448 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:16.880908 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:16.880878 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8"] Oct 30 18:09:16.884960 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:16.884942 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:16.917135 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:16.917104 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8"] Oct 30 18:09:17.004356 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.004326 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:09:17.004556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.004409 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" podUID="187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 18:09:17.075346 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.075309 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.075346 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.075349 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.075855 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.075366 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.075855 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.075459 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.075855 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.075553 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.075855 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.075572 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.075855 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.075601 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.075855 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.075655 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-264ph\" (UniqueName: \"kubernetes.io/projected/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-kube-api-access-264ph\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.075855 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.075721 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.176901 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.176826 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.176901 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.176856 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.177131 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.176904 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.177131 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.176958 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-264ph\" (UniqueName: \"kubernetes.io/projected/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-kube-api-access-264ph\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.177131 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.176996 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.177298 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.177163 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.177401 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.177382 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.177461 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.177427 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.177461 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.177438 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.177590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.177463 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.177590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.177551 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.177778 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.177756 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.177843 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.177760 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.178051 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.178017 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.179585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.179562 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.180170 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.180147 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.202328 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.202302 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.202577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.202560 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-264ph\" (UniqueName: \"kubernetes.io/projected/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-kube-api-access-264ph\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.499082 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.498977 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:17.833918 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.833890 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8"] Oct 30 18:09:17.836450 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:09:17.836419 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e7acbe_39d1_478a_aaf6_ad5f2b1091f8.slice/crio-ddd2a011a67431dbde720c2ed8fad2e78e86a73bc2a10b0e431f916d47f4cf9f WatchSource:0}: Error finding container ddd2a011a67431dbde720c2ed8fad2e78e86a73bc2a10b0e431f916d47f4cf9f: Status 404 returned error can't find the container with id ddd2a011a67431dbde720c2ed8fad2e78e86a73bc2a10b0e431f916d47f4cf9f Oct 30 18:09:17.838981 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.838949 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:17.839046 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.839017 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:17.839089 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.839046 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:17.993866 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.993832 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" event={"ID":"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8","Type":"ContainerStarted","Data":"69363d50945d8c790fa0bd6894e6c1d8e1d28f1e3abc4fc1c5ea7827a69ca489"} Oct 30 18:09:17.994020 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:17.993904 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" event={"ID":"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8","Type":"ContainerStarted","Data":"ddd2a011a67431dbde720c2ed8fad2e78e86a73bc2a10b0e431f916d47f4cf9f"} Oct 30 18:09:18.009911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:18.009886 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:18.010194 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:18.010173 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:18.010982 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:18.010958 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:09:18.040388 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:18.040357 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:18.041005 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:18.040966 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" podStartSLOduration=2.040953681 podStartE2EDuration="2.040953681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:18.037749014 +0000 UTC m=+1018.753318889" watchObservedRunningTime="2026-10-30 18:09:18.040953681 +0000 UTC m=+1018.756523581" Oct 30 18:09:18.506004 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:18.505961 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:18.506783 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:18.506746 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 18:09:18.506991 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:18.506957 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" podUID="c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 18:09:18.729245 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:18.729207 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:19.510528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:19.504739 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 18:09:19.510528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:19.504796 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" podUID="c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 18:09:19.742088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:19.742041 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:20.509968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:20.509929 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 18:09:20.511017 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:20.510985 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" podUID="c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 18:09:21.340964 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:21.340930 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:21.507535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:21.506938 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 18:09:21.507535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:21.507023 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" podUID="c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 18:09:21.941522 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:09:21.941462 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:09:22.511527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:22.510554 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 18:09:22.511527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:22.510610 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" podUID="c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 18:09:22.947597 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:22.947566 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:23.512636 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:23.512603 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 18:09:23.512931 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:23.512903 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" podUID="c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 18:09:24.499791 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:24.499757 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 18:09:24.500227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:24.499812 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" podUID="c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 18:09:24.742529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:24.727532 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:24.928455 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:24.928425 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:25.504636 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:25.504609 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:25.505043 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:25.504972 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:25.505704 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:25.505685 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:09:26.332248 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:26.332222 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:28.038434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:28.038399 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:29.623258 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:29.623211 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:31.344979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:31.344947 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:33.045486 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:33.045389 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:34.703672 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:34.703630 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:36.100410 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:36.100376 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:36.432777 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:36.432702 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:38.144137 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:38.144101 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:39.174531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:39.174489 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:09:39.863001 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:39.862969 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:41.973528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:41.973144 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:42.098535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:42.097614 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:42.132529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:42.131536 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:42.502530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:42.501883 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:42.594527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:42.593691 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:43.091651 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:43.091602 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:43.781652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:43.781592 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:44.041693 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:44.041588 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:44.050623 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:44.050579 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:44.094626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:44.094578 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:44.095673 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:44.095650 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:09:44.096606 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:44.095881 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:44.474580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:44.460369 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:44.587760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:44.587715 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:09:44.668608 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:44.668447 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:44.794703 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:44.794649 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:44.795896 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:44.795866 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:09:45.045722 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:45.045618 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:45.067758 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:45.046677 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:09:45.104543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:45.080693 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:45.104543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:45.103980 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:09:45.116328 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:45.113583 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:09:45.116328 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:45.114351 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:09:45.116328 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:45.115589 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:45.131108 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:45.131002 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:09:45.521590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:45.509111 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:09:45.557785 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:45.557729 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:45.609561 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:45.595615 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:45.609561 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:45.596567 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:09:45.654865 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:45.654834 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:45.845725 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:45.845634 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:09:46.051530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:46.046121 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:09:46.105690 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:46.105579 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:46.129534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:46.115986 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:46.130333 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:46.126176 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:46.151526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:46.147906 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:09:46.176528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:46.176059 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:09:46.297533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:46.288437 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:09:46.299376 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:46.299357 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:09:46.477755 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:46.477640 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:09:46.517584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:46.516202 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:46.517584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:46.516620 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:09:46.619062 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:46.619032 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:09:46.871343 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:46.871297 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:09:48.104295 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:48.101870 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:48.143590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:48.140963 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:48.143590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:48.141796 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:09:48.143590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:48.143528 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:09:48.143921 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:48.143798 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:09:48.233527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:48.223914 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:09:48.233527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:48.226182 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:09:48.305528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:48.294453 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:48.593556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:48.553571 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:48.593556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:48.553597 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 30 18:09:48.608527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:48.600818 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:48.634535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:48.622263 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:48.634535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:48.628965 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:48.964071 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:48.963976 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:09:48.964351 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:48.964304 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" podUID="b664f4ca-736e-4cb1-991d-985d1e6aff34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:49.021093 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:49.021046 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:09:49.043720 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:49.043668 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" podUID="4d6cd466-9f15-476d-9f04-6d03176f025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:49.142688 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:49.142648 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:09:49.147642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:49.147586 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:49.176718 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:49.169983 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:09:49.188580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:49.172209 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:09:49.188580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:49.185769 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:09:49.247618 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:49.240770 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:49.247618 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:49.241485 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:09:49.297419 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:49.297379 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:49.399831 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:49.398095 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:09:49.424006 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:49.423977 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:09:49.520381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:49.518900 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:09:49.539911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:49.539873 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:09:49.591031 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:49.587909 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:09:49.594619 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:49.594593 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:09:50.049318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:50.049279 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:51.744204 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:51.744161 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:51.931714 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:09:51.931678 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:09:53.237837 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:53.237797 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:53.400357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:53.400327 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:54.812242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:54.812199 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7"] Oct 30 18:09:54.817313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:54.817297 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:54.843443 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:54.843413 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7"] Oct 30 18:09:54.894547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:54.894491 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:54.894668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:54.894548 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:54.894668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:54.894569 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxwjm\" (UniqueName: \"kubernetes.io/projected/b6e43478-8cc1-4452-8bc8-fed14f127896-kube-api-access-wxwjm\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:54.894745 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:54.894663 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:54.894792 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:54.894780 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:54.894830 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:54.894806 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:54.894887 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:54.894876 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:54.894934 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:54.894923 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:54.894980 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:54.894970 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6e43478-8cc1-4452-8bc8-fed14f127896-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:54.944570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:54.944539 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:54.996404 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:54.996361 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:55.038358 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:55.038316 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6e43478-8cc1-4452-8bc8-fed14f127896-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:55.038622 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:55.038444 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:55.038622 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:55.038469 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:55.038622 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:55.038496 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxwjm\" (UniqueName: \"kubernetes.io/projected/b6e43478-8cc1-4452-8bc8-fed14f127896-kube-api-access-wxwjm\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:55.038622 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:55.038605 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:55.038867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:55.038793 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:55.038867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:55.038837 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:55.038978 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:55.038934 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:55.039406 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:54.997163 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:55.040393 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:55.040352 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6e43478-8cc1-4452-8bc8-fed14f127896-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:55.040859 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:55.040837 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:55.042133 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:55.042111 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:55.044148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:55.044119 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:55.059670 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:55.059634 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:55.060620 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:55.060544 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:55.065880 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:55.065192 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:55.089194 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:55.089058 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxwjm\" (UniqueName: \"kubernetes.io/projected/b6e43478-8cc1-4452-8bc8-fed14f127896-kube-api-access-wxwjm\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:55.130492 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:55.130464 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:55.457388 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:55.457360 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7"] Oct 30 18:09:55.458683 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:09:55.458656 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e43478_8cc1_4452_8bc8_fed14f127896.slice/crio-e0179e7723ba80fe74d0626fb7e382b3b74a5f773a12acfd12eb0e36e537a7d7 WatchSource:0}: Error finding container e0179e7723ba80fe74d0626fb7e382b3b74a5f773a12acfd12eb0e36e537a7d7: Status 404 returned error can't find the container with id e0179e7723ba80fe74d0626fb7e382b3b74a5f773a12acfd12eb0e36e537a7d7 Oct 30 18:09:55.461718 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:55.461688 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:55.461819 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:55.461788 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:55.461883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:55.461836 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:56.055713 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:56.055678 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" event={"ID":"b6e43478-8cc1-4452-8bc8-fed14f127896","Type":"ContainerStarted","Data":"3d67718133ad27c33c64da0b59e48736a8e2c418167704665f1bd379f42c597c"} Oct 30 18:09:56.056090 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:56.055739 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" event={"ID":"b6e43478-8cc1-4452-8bc8-fed14f127896","Type":"ContainerStarted","Data":"e0179e7723ba80fe74d0626fb7e382b3b74a5f773a12acfd12eb0e36e537a7d7"} Oct 30 18:09:56.109931 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:56.109888 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" podStartSLOduration=2.109874126 podStartE2EDuration="2.109874126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:56.106821561 +0000 UTC m=+1056.822391434" watchObservedRunningTime="2026-10-30 18:09:56.109874126 +0000 UTC m=+1056.825444064" Oct 30 18:09:56.130891 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:56.130868 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:09:56.132070 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:56.132047 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:09:56.132153 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:56.132099 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" podUID="b6e43478-8cc1-4452-8bc8-fed14f127896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 18:09:56.630067 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:56.630026 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:57.131049 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:57.131018 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:09:57.131536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:57.131078 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" podUID="b6e43478-8cc1-4452-8bc8-fed14f127896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 18:09:58.131488 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:58.131456 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:09:58.131974 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:58.131538 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" podUID="b6e43478-8cc1-4452-8bc8-fed14f127896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 18:09:58.245990 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:58.245960 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:59.131462 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:59.131426 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:09:59.131751 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:59.131480 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" podUID="b6e43478-8cc1-4452-8bc8-fed14f127896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 18:09:59.960527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:09:59.958191 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:00.130892 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:00.130855 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:10:00.131075 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:00.130920 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" podUID="b6e43478-8cc1-4452-8bc8-fed14f127896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 18:10:01.131712 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:01.131661 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:10:01.132151 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:01.131757 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" podUID="b6e43478-8cc1-4452-8bc8-fed14f127896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 18:10:01.643990 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:01.643954 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:02.131055 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:02.131020 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:10:02.131368 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:02.131332 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" podUID="b6e43478-8cc1-4452-8bc8-fed14f127896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 18:10:03.236208 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:03.236173 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:10:03.237569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:03.237548 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:10:03.238038 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:03.238022 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:10:03.349537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:03.348332 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:04.936128 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:04.936075 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:06.688530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:06.687685 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:08.348797 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:08.348760 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:09.935651 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:09.935610 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:11.561245 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:11.561210 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:13.362700 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:13.362669 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:13.852428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:13.852376 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:15.048606 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:15.048573 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:16.743055 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:16.743011 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:18.427941 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:18.427908 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:19.551417 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:19.551387 4603 scope.go:122] "RemoveContainer" containerID="63f915de7d4e62444545babb63dfa0ff476258dad47682955ffeb8eba7e2f435" Oct 30 18:10:20.040757 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:20.040727 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:21.642903 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:21.642865 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:21.785577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:21.785529 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:22.177996 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:10:22.177958 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:10:23.625801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:23.622193 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:25.387052 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:25.387012 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:27.192556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:27.192479 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:28.733160 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:28.733130 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:29.632069 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:29.632038 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:10:30.325180 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:30.325146 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:32.234391 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:32.234361 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:33.181802 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:33.181695 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:33.434426 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:33.434193 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:33.460812 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:33.460771 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:10:33.527682 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:33.527627 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:33.545637 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:33.545580 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:33.600349 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:33.600308 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:33.627785 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:33.624029 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:33.629435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:33.624365 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:10:33.943643 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:33.943604 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:10:33.943960 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:33.943930 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" podUID="b664f4ca-736e-4cb1-991d-985d1e6aff34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:34.095527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:34.076631 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:10:34.095527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:34.076700 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" podUID="187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:34.281527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:34.277662 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:10:34.281527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:34.277726 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" podUID="1d187b51-f5c6-4a63-8300-2f01184a15d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:34.281527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:34.278061 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:10:34.281527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:34.278101 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" podUID="d4a3adf6-fb23-4334-8bab-f800b4e8e089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:34.307529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:34.304164 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:10:34.307529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:34.304231 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" podUID="b6e43478-8cc1-4452-8bc8-fed14f127896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:34.486532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:34.472627 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:10:34.486532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:34.472994 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:10:34.511531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:34.487120 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:34.554581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:34.543418 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:34.554581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:34.544385 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:10:34.554581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:34.544440 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:10:34.554581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:34.546301 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:34.569492 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:34.554988 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:10:34.631703 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:34.631645 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Oct 30 18:10:34.676528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:34.638593 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:10:34.745543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:34.731676 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:10:34.745543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:34.641633 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:10:34.754531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:34.746839 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:10:34.786530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:34.641671 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:34.798529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:34.791079 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:10:35.161736 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:35.161036 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:10:35.511586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:35.505684 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:10:35.532260 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:35.530980 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:10:35.532260 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:35.532068 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:35.553151 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:35.553021 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:10:35.554252 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:35.553902 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:35.767653 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:35.767542 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:10:35.800529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:35.798034 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:10:35.861530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:35.851258 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:10:36.222529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:36.210841 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:10:36.242531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:36.223485 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:10:36.515583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:36.511769 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:36.552337 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:36.551316 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:36.556887 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:36.556599 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:10:36.578685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:36.562598 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:36.771589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:36.770197 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:36.801529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:36.798340 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:36.891530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:36.852615 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:37.208703 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:37.208598 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:37.224645 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:37.224594 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:10:37.347655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:37.347606 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:37.512545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:37.512424 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:37.556642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:37.556592 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:37.557593 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:37.557108 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:37.564958 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:37.564583 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:37.794557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:37.794491 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:10:37.821532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:37.818634 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:37.855628 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:37.855581 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:38.108437 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:38.107167 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:38.358581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:38.357315 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:38.492397 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:38.492217 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:38.519883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:38.518649 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:38.519883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:38.519035 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 30 18:10:38.604355 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:38.577007 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:38.604355 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:38.578866 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:38.612584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:38.607354 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:38.776986 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:38.776686 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Oct 30 18:10:38.859256 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:38.850485 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:10:38.859256 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:38.850772 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:10:38.861067 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:38.861031 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:39.359646 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:39.359600 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:39.520664 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:39.520587 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:39.594336 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:39.594296 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:39.615942 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:39.615850 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:10:39.675539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:39.660020 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:39.854539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:39.851622 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:39.854539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:39.852086 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:10:39.861756 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:39.861721 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:40.162826 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:40.162791 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:10:40.163096 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:40.163068 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" podUID="e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:40.248294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:40.248261 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:40.293813 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:40.293755 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:40.365981 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:40.365917 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:40.543575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:40.524072 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:10:40.617576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:40.611156 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:40.617576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:40.616873 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:40.690531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:40.686203 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:40.722547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:40.721810 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:10:40.859652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:40.859547 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:40.860240 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:40.860205 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:41.020201 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:41.019022 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:10:41.046052 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:41.046013 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:10:41.048469 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:41.048444 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:10:41.051278 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:41.051255 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:10:41.053157 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:41.053137 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:10:41.054752 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:41.054732 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:10:41.062844 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:41.062819 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:10:41.989160 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:41.989121 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:43.658527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:43.640217 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:45.258069 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:45.258027 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:46.859863 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:46.859830 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:48.638774 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:48.638742 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:50.436995 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:50.436958 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:51.930454 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:10:51.930422 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:10:52.202883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:52.202785 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:53.851841 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:53.851804 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:55.263483 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:55.263451 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:55.403755 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:55.403724 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:55.686117 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:55.686080 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:57.348127 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:57.348099 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:58.959949 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:10:58.959918 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:00.753497 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:00.753461 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:02.512636 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:02.512542 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:04.144853 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:04.144819 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:05.854281 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:05.854245 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:06.961777 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:06.961742 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:07.677413 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:07.677376 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:09.254914 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:09.251183 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:10.847070 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:10.847037 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:12.445368 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:12.445335 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:12.978002 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:12.977969 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:13.223942 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:13.223909 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:13.371667 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:13.371638 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:13.963056 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:13.963019 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:14.153315 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:14.153280 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:14.345534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:14.332112 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:14.506047 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:14.506000 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:15.951656 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:15.951605 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:17.535642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:17.535614 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:19.219402 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:19.219371 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:21.015285 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:21.015257 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:21.559493 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:21.559461 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:11:21.827862 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:21.827786 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:21.930399 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:11:21.930364 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:11:22.227208 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.227123 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq"] Oct 30 18:11:22.232084 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.232060 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.250052 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.250017 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c6b2120f-openshift-default-dockercfg-xt8vs\"" Oct 30 18:11:22.267456 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.267430 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq"] Oct 30 18:11:22.311124 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.311096 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-workload-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.311258 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.311151 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-envoy\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.311258 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.311177 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-workload-certs\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.311334 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.311282 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-data\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.311334 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.311315 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx4xj\" (UniqueName: \"kubernetes.io/projected/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-kube-api-access-gx4xj\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.311400 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.311389 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-token\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.311434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.311428 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-podinfo\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.311473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.311454 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istiod-ca-cert\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.311532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.311471 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-credential-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.314409 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.314379 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:22.448539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.434825 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-workload-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.448936 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.448906 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-envoy\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.463531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.449068 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-workload-certs\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.481565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.437431 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-workload-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.481565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.453376 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-workload-certs\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.481565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.478536 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-data\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.481565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.478613 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx4xj\" (UniqueName: \"kubernetes.io/projected/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-kube-api-access-gx4xj\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.481565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.478917 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-token\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.481565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.478995 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-podinfo\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.481565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.479046 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istiod-ca-cert\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.481565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.479079 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-credential-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.481565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.479574 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-data\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.481565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.480368 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-envoy\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.481565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.481048 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istiod-ca-cert\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.486528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.486335 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-credential-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.513527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.509567 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx4xj\" (UniqueName: \"kubernetes.io/projected/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-kube-api-access-gx4xj\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.580525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.547997 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:22.580525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.548979 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-podinfo\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.612531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.606266 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-token\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:22.712751 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.712718 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:22.874285 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:22.874253 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:23.230965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:23.230941 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq"] Oct 30 18:11:23.231626 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:11:23.231606 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a7d3cd_7df0_41c6_9063_f2caf0e934d4.slice/crio-9226b5319566a1063de47071898ce2947388e43c185531382cb8a9b7afebebbc WatchSource:0}: Error finding container 9226b5319566a1063de47071898ce2947388e43c185531382cb8a9b7afebebbc: Status 404 returned error can't find the container with id 9226b5319566a1063de47071898ce2947388e43c185531382cb8a9b7afebebbc Oct 30 18:11:23.234060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:23.234020 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:23.234150 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:23.234092 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:23.234150 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:23.234123 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:23.682291 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:23.682237 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" event={"ID":"02a7d3cd-7df0-41c6-9063-f2caf0e934d4","Type":"ContainerStarted","Data":"cd9faa3b3a83d54fc93fc5f072721bd4c1d303273fc7835188352f5b849a5827"} Oct 30 18:11:23.682635 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:23.682609 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" event={"ID":"02a7d3cd-7df0-41c6-9063-f2caf0e934d4","Type":"ContainerStarted","Data":"9226b5319566a1063de47071898ce2947388e43c185531382cb8a9b7afebebbc"} Oct 30 18:11:23.764935 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:23.764878 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" podStartSLOduration=1.764858245 podStartE2EDuration="1.764858245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:23.747914019 +0000 UTC m=+1144.463483895" watchObservedRunningTime="2026-10-30 18:11:23.764858245 +0000 UTC m=+1144.480428121" Oct 30 18:11:23.879659 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:23.879623 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:24.791053 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:24.791020 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:24.952562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:24.940823 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:25.504523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:25.503581 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:25.520528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:25.518809 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:25.567904 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:25.567852 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:25.570752 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:25.569885 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:25.714949 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:25.714919 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:25.721823 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:25.721784 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:11:25.840530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:25.832681 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:25.866529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:25.857933 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:26.377527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:26.373048 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:11:26.567541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:26.565926 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:11:26.567541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:26.565984 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" podUID="c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:26.591709 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:26.590450 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:26.591990 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:26.591948 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:26.593279 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:26.593258 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:11:26.817358 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:26.817324 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:26.832968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:26.832920 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:26.835171 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:26.835132 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:11:26.898536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:26.858742 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:26.898536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:26.863255 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:11:26.964050 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:26.964013 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:11:27.011779 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:27.011651 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:27.224730 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:27.224637 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:27.301751 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:27.299697 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:11:27.301751 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:27.299763 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" podUID="d5567046-aa24-421a-8339-fdd9cf56e62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:27.356527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:27.349768 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:27.390817 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:27.375604 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:27.401642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:27.401612 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:11:27.527704 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:27.527666 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:27.556852 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:27.556809 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:27.595685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:27.595642 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:27.596538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:27.596438 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:11:27.802424 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:27.802335 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:11:27.834733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:27.834690 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:11:27.897938 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:27.862633 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:11:27.965260 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:27.965217 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:11:27.966390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:27.966365 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:11:28.193106 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:28.192988 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": context deadline exceeded" Oct 30 18:11:28.393768 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:28.393724 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:28.436857 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:28.436595 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 30 18:11:28.533594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:28.532495 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:28.554530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:28.548452 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:11:28.583534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:28.582660 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:28.583534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:28.583228 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:11:28.650860 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:28.646628 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:28.650860 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:28.646741 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:28.650860 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:28.650152 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:28.931875 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:28.931767 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:11:28.932526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:28.932473 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:11:28.985980 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:28.985937 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:11:29.124654 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:29.124615 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:11:29.124936 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:29.124906 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" podUID="22e0b280-2b65-4fad-8344-67049190d108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:29.396530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:29.394601 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:11:29.535531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:29.535458 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:29.600627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:29.600581 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:29.656582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:29.655734 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:29.656747 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:29.656725 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:29.768129 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:29.768089 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:29.936760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:29.933052 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:29.936760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:29.933525 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:11:29.937254 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:29.937226 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:30.011656 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:30.011609 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:30.012734 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:30.012701 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:11:30.012847 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:30.012757 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" podUID="81f49829-73ed-4760-9444-f1793861eba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:30.077759 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:30.077614 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:11:30.077759 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:30.077691 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" podUID="571fc9dd-71e3-4fa7-b1af-e74f0a96654d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:30.321405 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:30.321353 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:30.417616 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:30.417573 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:30.559525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:30.559246 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:11:30.637613 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:30.636624 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:11:30.665526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:30.661261 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:30.666840 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:30.666791 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:30.943751 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:30.943630 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:30.944382 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:30.944348 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:11:31.034238 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:31.030756 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:11:31.131520 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:31.130471 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:31.439799 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:31.439747 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:31.560648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:31.560595 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:11:31.641532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:31.640653 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:31.666089 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:31.666029 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:31.667544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:31.667485 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:31.945673 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:31.944661 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:31.962490 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:31.962450 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:11:32.032669 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:32.032094 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:11:32.458570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:32.449697 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:11:32.566564 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:32.566495 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:32.652151 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:32.652111 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:32.672638 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:32.672577 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:11:32.674883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:32.674827 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:32.948695 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:32.948644 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:32.973544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:32.965595 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:33.047493 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:33.046209 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:11:33.451675 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:33.451627 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:33.511528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:33.508338 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:11:33.529027 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:33.508495 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" podUID="ec3c95aa-215c-4f15-8dc8-bc0beb954236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:33.531460 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:33.531425 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:33.650857 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:33.648583 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:33.661714 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:33.661658 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:11:33.682572 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:33.674601 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:11:33.690613 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:33.690282 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:33.952154 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:33.951622 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:33.976531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:33.976411 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:34.048631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:34.048584 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:34.126526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:34.096097 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:11:34.126526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:34.096168 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" podUID="187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:34.290593 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:34.282615 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:11:34.290593 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:34.282679 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" podUID="b6e43478-8cc1-4452-8bc8-fed14f127896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:34.477622 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:34.477574 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:34.514994 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:34.514967 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:11:34.631542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:34.627966 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Oct 30 18:11:34.688307 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:34.688259 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:34.702278 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:34.701680 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:34.706737 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:34.706179 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:11:34.758260 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:34.754778 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:11:34.794552 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:34.794497 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:11:34.808567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:34.807058 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:11:34.808567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:34.808223 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:11:34.809363 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:34.808931 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:11:34.811440 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:34.811418 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:11:34.812378 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:34.812335 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:11:34.814958 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:34.814936 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:11:35.152857 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:35.152825 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:36.313648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:36.313621 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:36.830942 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:36.830911 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:38.607699 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:38.607658 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:40.306026 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:40.305990 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:42.001601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:42.001560 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:42.161538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:42.160162 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:42.299903 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:42.299868 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:42.444811 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:42.444765 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:43.754353 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:43.754314 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:45.538516 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:45.538457 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:47.163897 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:47.163859 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:48.486061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:48.486024 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:49.010789 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:49.010757 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:49.318868 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:49.318842 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:49.478878 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:49.478842 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:50.635479 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:50.635447 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:51.931855 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:11:51.931809 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:11:52.348136 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:52.348100 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:53.076701 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.076670 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:53.492972 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.492891 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl"] Oct 30 18:11:53.498150 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.498124 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.517145 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.517120 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c5891f-openshift-default-dockercfg-hmphn\"" Oct 30 18:11:53.532116 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.532092 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl"] Oct 30 18:11:53.568937 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.568907 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b162d38e-8491-4a60-8e12-c6338131cd03-istio-podinfo\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.569085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.568950 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-istio-data\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.569085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.569012 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-credential-socket\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.569194 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.569110 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-workload-certs\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.569194 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.569132 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-istio-envoy\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.569297 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.569201 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b162d38e-8491-4a60-8e12-c6338131cd03-istio-token\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.569297 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.569271 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b162d38e-8491-4a60-8e12-c6338131cd03-istiod-ca-cert\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.569398 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.569310 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-workload-socket\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.569450 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.569397 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9wd7\" (UniqueName: \"kubernetes.io/projected/b162d38e-8491-4a60-8e12-c6338131cd03-kube-api-access-k9wd7\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.656530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.652694 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:53.691529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.670622 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-credential-socket\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.692359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.692334 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-workload-certs\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.692558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.692539 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-istio-envoy\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.692717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.692699 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b162d38e-8491-4a60-8e12-c6338131cd03-istio-token\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.692932 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.692915 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b162d38e-8491-4a60-8e12-c6338131cd03-istiod-ca-cert\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.694673 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.694653 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-workload-socket\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.703825 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.703802 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9wd7\" (UniqueName: \"kubernetes.io/projected/b162d38e-8491-4a60-8e12-c6338131cd03-kube-api-access-k9wd7\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.704114 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.694540 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b162d38e-8491-4a60-8e12-c6338131cd03-istiod-ca-cert\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.709603 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.693272 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-workload-certs\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.709772 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.695037 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-workload-socket\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.709859 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.703810 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-istio-envoy\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.709939 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.692186 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-credential-socket\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.710018 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.704103 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b162d38e-8491-4a60-8e12-c6338131cd03-istio-podinfo\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.727317 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.727289 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-istio-data\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.727949 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.727926 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-istio-data\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.741284 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.741251 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b162d38e-8491-4a60-8e12-c6338131cd03-istio-podinfo\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.759573 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.756243 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9wd7\" (UniqueName: \"kubernetes.io/projected/b162d38e-8491-4a60-8e12-c6338131cd03-kube-api-access-k9wd7\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.767609 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.758568 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b162d38e-8491-4a60-8e12-c6338131cd03-istio-token\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.824527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.823798 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:53.908588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:53.908553 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:54.151642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:54.151344 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:54.290874 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:54.290816 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl"] Oct 30 18:11:54.305143 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:54.304856 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:54.305143 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:54.305022 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:54.305143 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:54.305111 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:54.975665 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:54.975630 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" event={"ID":"b162d38e-8491-4a60-8e12-c6338131cd03","Type":"ContainerStarted","Data":"9368e601f948256dc178db7455f99ecbd3896b8c9ef80ee1382f6c80eb434483"} Oct 30 18:11:54.975862 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:54.975713 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" event={"ID":"b162d38e-8491-4a60-8e12-c6338131cd03","Type":"ContainerStarted","Data":"f206ebfae50ef2582b2d6e3e7520cce1a7dd6372c7ef42c8d8999e8b409b9e63"} Oct 30 18:11:55.031723 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:55.031681 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" podStartSLOduration=2.031666234 podStartE2EDuration="2.031666234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:55.031106446 +0000 UTC m=+1175.746676320" watchObservedRunningTime="2026-10-30 18:11:55.031666234 +0000 UTC m=+1175.747236135" Oct 30 18:11:55.805525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:55.802282 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:55.829659 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:55.825456 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:55.843493 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:55.843427 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:55.983119 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:55.983095 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:55.983954 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:55.983936 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:11:57.456556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:57.456528 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:57.600741 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:57.600651 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:59.255322 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:59.255283 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:59.397575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:11:59.397543 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:00.492871 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:00.492838 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:00.634101 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:00.634068 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:01.029653 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:01.029627 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:02.641387 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:02.641350 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:04.231002 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:04.230970 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:05.931890 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:05.931849 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:07.567727 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:07.567687 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:09.212569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:09.212534 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:09.378163 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:09.378130 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:09.524828 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:09.524790 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:10.502035 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:10.501998 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522"] Oct 30 18:12:10.502465 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:10.502219 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" podUID="5bf7d070-07da-4ad5-8464-e39192f8e23e" containerName="istio-proxy" containerID="cri-o://2ec99e2ec8f1eda19858be1f8b34e7895b4a91b5daf5d7bf9a3208b65516bb93" gracePeriod=30 Oct 30 18:12:10.514359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:10.514332 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh"] Oct 30 18:12:10.517282 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:10.517256 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr"] Oct 30 18:12:10.517443 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:10.517423 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" podUID="049906ec-ccdb-45e9-8fe9-9f50150fc7f6" containerName="istio-proxy" containerID="cri-o://f18a4dbda2fc46a13c70f70884ce40718e12029125e3b680bd57fa84f651a413" gracePeriod=30 Oct 30 18:12:10.948924 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:10.948892 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:11.113702 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:11.113608 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" podUID="53429703-95ae-4f02-8c26-a9a296c56a05" containerName="istio-proxy" containerID="cri-o://7e2406130b23b97ceafe24ac4af504bcc85fe97a8a0703755fd7c01cbadefa7e" gracePeriod=30 Oct 30 18:12:12.622445 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:12.622412 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:14.072782 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:14.072733 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:14.222047 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:14.221967 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:14.373367 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:14.373337 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:15.442191 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.442155 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qmzkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:12:15.442616 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.442213 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" podUID="53429703-95ae-4f02-8c26-a9a296c56a05" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:12:15.802786 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.802758 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:12:15.835541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.835516 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 18:12:15.868396 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.868341 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-token\" (UniqueName: \"kubernetes.io/projected/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-token\") pod \"5bf7d070-07da-4ad5-8464-e39192f8e23e\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " Oct 30 18:12:15.868583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.868414 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-envoy\") pod \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " Oct 30 18:12:15.868583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.868456 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-data\") pod \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " Oct 30 18:12:15.868583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.868486 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-workload-certs\") pod \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " Oct 30 18:12:15.868583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.868570 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-workload-socket\") pod \"5bf7d070-07da-4ad5-8464-e39192f8e23e\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " Oct 30 18:12:15.868801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.868616 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-token\" (UniqueName: \"kubernetes.io/projected/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-token\") pod \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " Oct 30 18:12:15.868801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.868722 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-kube-api-access-hjwz8\" (UniqueName: \"kubernetes.io/projected/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-kube-api-access-hjwz8\") pod \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " Oct 30 18:12:15.868801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.868756 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-credential-socket\") pod \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " Oct 30 18:12:15.868959 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.868899 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-podinfo\") pod \"5bf7d070-07da-4ad5-8464-e39192f8e23e\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " Oct 30 18:12:15.869011 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.868979 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bf7d070-07da-4ad5-8464-e39192f8e23e-kube-api-access-7cxhw\" (UniqueName: \"kubernetes.io/projected/5bf7d070-07da-4ad5-8464-e39192f8e23e-kube-api-access-7cxhw\") pod \"5bf7d070-07da-4ad5-8464-e39192f8e23e\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " Oct 30 18:12:15.869063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.869040 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bf7d070-07da-4ad5-8464-e39192f8e23e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bf7d070-07da-4ad5-8464-e39192f8e23e-istiod-ca-cert\") pod \"5bf7d070-07da-4ad5-8464-e39192f8e23e\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " Oct 30 18:12:15.869119 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.869093 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istiod-ca-cert\") pod \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " Oct 30 18:12:15.869172 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.869152 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-credential-socket\") pod \"5bf7d070-07da-4ad5-8464-e39192f8e23e\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " Oct 30 18:12:15.869225 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.869184 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-workload-certs\") pod \"5bf7d070-07da-4ad5-8464-e39192f8e23e\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " Oct 30 18:12:15.869278 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.869231 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-podinfo\") pod \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " Oct 30 18:12:15.869333 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.869273 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-data\") pod \"5bf7d070-07da-4ad5-8464-e39192f8e23e\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " Oct 30 18:12:15.869333 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.869292 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-workload-socket" pod "5bf7d070-07da-4ad5-8464-e39192f8e23e" (UID: "5bf7d070-07da-4ad5-8464-e39192f8e23e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:12:15.869333 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.869306 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-workload-socket\") pod \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\" (UID: \"049906ec-ccdb-45e9-8fe9-9f50150fc7f6\") " Oct 30 18:12:15.869472 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.869337 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-envoy\") pod \"5bf7d070-07da-4ad5-8464-e39192f8e23e\" (UID: \"5bf7d070-07da-4ad5-8464-e39192f8e23e\") " Oct 30 18:12:15.869795 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.869768 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-data" pod "049906ec-ccdb-45e9-8fe9-9f50150fc7f6" (UID: "049906ec-ccdb-45e9-8fe9-9f50150fc7f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:12:15.870061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.870040 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:15.870061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.870066 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:15.870582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.870231 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-workload-certs" pod "049906ec-ccdb-45e9-8fe9-9f50150fc7f6" (UID: "049906ec-ccdb-45e9-8fe9-9f50150fc7f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:12:15.870582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.870467 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-credential-socket" pod "049906ec-ccdb-45e9-8fe9-9f50150fc7f6" (UID: "049906ec-ccdb-45e9-8fe9-9f50150fc7f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:12:15.870582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.870550 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bf7d070-07da-4ad5-8464-e39192f8e23e-istiod-ca-cert" pod "5bf7d070-07da-4ad5-8464-e39192f8e23e" (UID: "5bf7d070-07da-4ad5-8464-e39192f8e23e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:12:15.870941 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.870857 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-workload-certs" pod "5bf7d070-07da-4ad5-8464-e39192f8e23e" (UID: "5bf7d070-07da-4ad5-8464-e39192f8e23e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:12:15.871287 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.871143 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istiod-ca-cert" pod "049906ec-ccdb-45e9-8fe9-9f50150fc7f6" (UID: "049906ec-ccdb-45e9-8fe9-9f50150fc7f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:12:15.871380 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.871358 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-credential-socket" pod "5bf7d070-07da-4ad5-8464-e39192f8e23e" (UID: "5bf7d070-07da-4ad5-8464-e39192f8e23e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:12:15.871581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.871555 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-data" pod "5bf7d070-07da-4ad5-8464-e39192f8e23e" (UID: "5bf7d070-07da-4ad5-8464-e39192f8e23e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:12:15.871666 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.871592 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-workload-socket" pod "049906ec-ccdb-45e9-8fe9-9f50150fc7f6" (UID: "049906ec-ccdb-45e9-8fe9-9f50150fc7f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:12:15.873538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.873491 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-token" pod "5bf7d070-07da-4ad5-8464-e39192f8e23e" (UID: "5bf7d070-07da-4ad5-8464-e39192f8e23e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:12:15.873645 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.873575 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf7d070-07da-4ad5-8464-e39192f8e23e-kube-api-access-7cxhw" pod "5bf7d070-07da-4ad5-8464-e39192f8e23e" (UID: "5bf7d070-07da-4ad5-8464-e39192f8e23e"). InnerVolumeSpecName "kube-api-access-7cxhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:12:15.873645 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.873629 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-envoy" pod "049906ec-ccdb-45e9-8fe9-9f50150fc7f6" (UID: "049906ec-ccdb-45e9-8fe9-9f50150fc7f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:12:15.873931 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.873908 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-envoy" pod "5bf7d070-07da-4ad5-8464-e39192f8e23e" (UID: "5bf7d070-07da-4ad5-8464-e39192f8e23e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:12:15.873982 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.873923 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-podinfo" pod "5bf7d070-07da-4ad5-8464-e39192f8e23e" (UID: "5bf7d070-07da-4ad5-8464-e39192f8e23e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:12:15.874775 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.874746 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-podinfo" pod "049906ec-ccdb-45e9-8fe9-9f50150fc7f6" (UID: "049906ec-ccdb-45e9-8fe9-9f50150fc7f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:12:15.874961 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.874941 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-kube-api-access-hjwz8" pod "049906ec-ccdb-45e9-8fe9-9f50150fc7f6" (UID: "049906ec-ccdb-45e9-8fe9-9f50150fc7f6"). InnerVolumeSpecName "kube-api-access-hjwz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:12:15.875548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.875495 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-token" pod "049906ec-ccdb-45e9-8fe9-9f50150fc7f6" (UID: "049906ec-ccdb-45e9-8fe9-9f50150fc7f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:12:15.951341 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.951312 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:15.971147 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.971122 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:15.971147 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.971144 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjwz8\" (UniqueName: \"kubernetes.io/projected/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-kube-api-access-hjwz8\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:15.971308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.971153 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:15.971308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.971163 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:15.971308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.971172 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cxhw\" (UniqueName: \"kubernetes.io/projected/5bf7d070-07da-4ad5-8464-e39192f8e23e-kube-api-access-7cxhw\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:15.971308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.971180 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bf7d070-07da-4ad5-8464-e39192f8e23e-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:15.971308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.971188 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:15.971308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.971196 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:15.971308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.971203 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:15.971308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.971212 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:15.971308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.971219 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:15.971308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.971227 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:15.971308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.971234 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:15.971308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.971243 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bf7d070-07da-4ad5-8464-e39192f8e23e-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:15.971308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.971250 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:15.971308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:15.971257 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049906ec-ccdb-45e9-8fe9-9f50150fc7f6-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:16.160872 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.160781 4603 generic.go:356] "Generic (PLEG): container finished" podID="049906ec-ccdb-45e9-8fe9-9f50150fc7f6" containerID="f18a4dbda2fc46a13c70f70884ce40718e12029125e3b680bd57fa84f651a413" exitCode=0 Oct 30 18:12:16.161156 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.161136 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" Oct 30 18:12:16.161976 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.161685 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" event={"ID":"049906ec-ccdb-45e9-8fe9-9f50150fc7f6","Type":"ContainerDied","Data":"f18a4dbda2fc46a13c70f70884ce40718e12029125e3b680bd57fa84f651a413"} Oct 30 18:12:16.162092 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.162046 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr" event={"ID":"049906ec-ccdb-45e9-8fe9-9f50150fc7f6","Type":"ContainerDied","Data":"de2dc335016ad12f1d82771f8508728b6483fdf340ff24fc1e1c212adadd7d37"} Oct 30 18:12:16.162171 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.162107 4603 scope.go:122] "RemoveContainer" containerID="f18a4dbda2fc46a13c70f70884ce40718e12029125e3b680bd57fa84f651a413" Oct 30 18:12:16.167375 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.167340 4603 generic.go:356] "Generic (PLEG): container finished" podID="5bf7d070-07da-4ad5-8464-e39192f8e23e" containerID="2ec99e2ec8f1eda19858be1f8b34e7895b4a91b5daf5d7bf9a3208b65516bb93" exitCode=0 Oct 30 18:12:16.167463 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.167376 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" event={"ID":"5bf7d070-07da-4ad5-8464-e39192f8e23e","Type":"ContainerDied","Data":"2ec99e2ec8f1eda19858be1f8b34e7895b4a91b5daf5d7bf9a3208b65516bb93"} Oct 30 18:12:16.167463 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.167444 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" event={"ID":"5bf7d070-07da-4ad5-8464-e39192f8e23e","Type":"ContainerDied","Data":"b67cf4b1f3224c667801624ea198242013f3164d33bd8a7362a3e4cd0ff77720"} Oct 30 18:12:16.167717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.167389 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522" Oct 30 18:12:16.177270 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.177248 4603 scope.go:122] "RemoveContainer" containerID="f18a4dbda2fc46a13c70f70884ce40718e12029125e3b680bd57fa84f651a413" Oct 30 18:12:16.177574 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:12:16.177555 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f18a4dbda2fc46a13c70f70884ce40718e12029125e3b680bd57fa84f651a413\": container with ID starting with f18a4dbda2fc46a13c70f70884ce40718e12029125e3b680bd57fa84f651a413 not found: ID does not exist" containerID="f18a4dbda2fc46a13c70f70884ce40718e12029125e3b680bd57fa84f651a413" Oct 30 18:12:16.177667 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.177582 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18a4dbda2fc46a13c70f70884ce40718e12029125e3b680bd57fa84f651a413"} err="failed to get container status \"f18a4dbda2fc46a13c70f70884ce40718e12029125e3b680bd57fa84f651a413\": rpc error: code = NotFound desc = could not find container \"f18a4dbda2fc46a13c70f70884ce40718e12029125e3b680bd57fa84f651a413\": container with ID starting with f18a4dbda2fc46a13c70f70884ce40718e12029125e3b680bd57fa84f651a413 not found: ID does not exist" Oct 30 18:12:16.177667 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.177599 4603 scope.go:122] "RemoveContainer" containerID="2ec99e2ec8f1eda19858be1f8b34e7895b4a91b5daf5d7bf9a3208b65516bb93" Oct 30 18:12:16.235585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.235567 4603 scope.go:122] "RemoveContainer" containerID="2ec99e2ec8f1eda19858be1f8b34e7895b4a91b5daf5d7bf9a3208b65516bb93" Oct 30 18:12:16.235880 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:12:16.235860 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ec99e2ec8f1eda19858be1f8b34e7895b4a91b5daf5d7bf9a3208b65516bb93\": container with ID starting with 2ec99e2ec8f1eda19858be1f8b34e7895b4a91b5daf5d7bf9a3208b65516bb93 not found: ID does not exist" containerID="2ec99e2ec8f1eda19858be1f8b34e7895b4a91b5daf5d7bf9a3208b65516bb93" Oct 30 18:12:16.235954 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.235891 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ec99e2ec8f1eda19858be1f8b34e7895b4a91b5daf5d7bf9a3208b65516bb93"} err="failed to get container status \"2ec99e2ec8f1eda19858be1f8b34e7895b4a91b5daf5d7bf9a3208b65516bb93\": rpc error: code = NotFound desc = could not find container \"2ec99e2ec8f1eda19858be1f8b34e7895b4a91b5daf5d7bf9a3208b65516bb93\": container with ID starting with 2ec99e2ec8f1eda19858be1f8b34e7895b4a91b5daf5d7bf9a3208b65516bb93 not found: ID does not exist" Oct 30 18:12:16.270085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.270053 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522"] Oct 30 18:12:16.282079 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.282052 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x4522"] Oct 30 18:12:16.311989 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.311962 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr"] Oct 30 18:12:16.317062 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.316976 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fhwtr"] Oct 30 18:12:16.365533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.365511 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 18:12:16.377277 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.377257 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-istio-data\") pod \"53429703-95ae-4f02-8c26-a9a296c56a05\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " Oct 30 18:12:16.377367 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.377344 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-istio-envoy\") pod \"53429703-95ae-4f02-8c26-a9a296c56a05\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " Oct 30 18:12:16.377421 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.377373 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53429703-95ae-4f02-8c26-a9a296c56a05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53429703-95ae-4f02-8c26-a9a296c56a05-istio-podinfo\") pod \"53429703-95ae-4f02-8c26-a9a296c56a05\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " Oct 30 18:12:16.377477 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.377456 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-workload-socket\") pod \"53429703-95ae-4f02-8c26-a9a296c56a05\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " Oct 30 18:12:16.377556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.377479 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-istio-data" pod "53429703-95ae-4f02-8c26-a9a296c56a05" (UID: "53429703-95ae-4f02-8c26-a9a296c56a05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:12:16.377556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.377492 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-workload-certs\") pod \"53429703-95ae-4f02-8c26-a9a296c56a05\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " Oct 30 18:12:16.377556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.377537 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53429703-95ae-4f02-8c26-a9a296c56a05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53429703-95ae-4f02-8c26-a9a296c56a05-istiod-ca-cert\") pod \"53429703-95ae-4f02-8c26-a9a296c56a05\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " Oct 30 18:12:16.377701 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.377627 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53429703-95ae-4f02-8c26-a9a296c56a05-kube-api-access-d8dpn\" (UniqueName: \"kubernetes.io/projected/53429703-95ae-4f02-8c26-a9a296c56a05-kube-api-access-d8dpn\") pod \"53429703-95ae-4f02-8c26-a9a296c56a05\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " Oct 30 18:12:16.377701 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.377667 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53429703-95ae-4f02-8c26-a9a296c56a05-istio-token\" (UniqueName: \"kubernetes.io/projected/53429703-95ae-4f02-8c26-a9a296c56a05-istio-token\") pod \"53429703-95ae-4f02-8c26-a9a296c56a05\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " Oct 30 18:12:16.377814 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.377721 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-credential-socket\") pod \"53429703-95ae-4f02-8c26-a9a296c56a05\" (UID: \"53429703-95ae-4f02-8c26-a9a296c56a05\") " Oct 30 18:12:16.377814 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.377747 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-workload-socket" pod "53429703-95ae-4f02-8c26-a9a296c56a05" (UID: "53429703-95ae-4f02-8c26-a9a296c56a05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:12:16.377814 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.377786 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-workload-certs" pod "53429703-95ae-4f02-8c26-a9a296c56a05" (UID: "53429703-95ae-4f02-8c26-a9a296c56a05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:12:16.378074 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.378052 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-credential-socket" pod "53429703-95ae-4f02-8c26-a9a296c56a05" (UID: "53429703-95ae-4f02-8c26-a9a296c56a05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:12:16.378180 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.378100 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53429703-95ae-4f02-8c26-a9a296c56a05-istiod-ca-cert" pod "53429703-95ae-4f02-8c26-a9a296c56a05" (UID: "53429703-95ae-4f02-8c26-a9a296c56a05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:12:16.378612 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.378587 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:16.378696 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.378616 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:16.378696 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.378632 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53429703-95ae-4f02-8c26-a9a296c56a05-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:16.378696 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.378646 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:16.378696 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.378669 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:16.379774 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.379752 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-istio-envoy" pod "53429703-95ae-4f02-8c26-a9a296c56a05" (UID: "53429703-95ae-4f02-8c26-a9a296c56a05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:12:16.379854 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.379787 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53429703-95ae-4f02-8c26-a9a296c56a05-istio-podinfo" pod "53429703-95ae-4f02-8c26-a9a296c56a05" (UID: "53429703-95ae-4f02-8c26-a9a296c56a05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:12:16.379915 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.379897 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53429703-95ae-4f02-8c26-a9a296c56a05-kube-api-access-d8dpn" pod "53429703-95ae-4f02-8c26-a9a296c56a05" (UID: "53429703-95ae-4f02-8c26-a9a296c56a05"). InnerVolumeSpecName "kube-api-access-d8dpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:12:16.380044 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.380024 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53429703-95ae-4f02-8c26-a9a296c56a05-istio-token" pod "53429703-95ae-4f02-8c26-a9a296c56a05" (UID: "53429703-95ae-4f02-8c26-a9a296c56a05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:12:16.480740 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.480670 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53429703-95ae-4f02-8c26-a9a296c56a05-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:16.480740 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.480693 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53429703-95ae-4f02-8c26-a9a296c56a05-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:16.480740 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.480706 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8dpn\" (UniqueName: \"kubernetes.io/projected/53429703-95ae-4f02-8c26-a9a296c56a05-kube-api-access-d8dpn\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:16.480740 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:16.480719 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53429703-95ae-4f02-8c26-a9a296c56a05-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:12:17.175077 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:17.175044 4603 generic.go:356] "Generic (PLEG): container finished" podID="53429703-95ae-4f02-8c26-a9a296c56a05" containerID="7e2406130b23b97ceafe24ac4af504bcc85fe97a8a0703755fd7c01cbadefa7e" exitCode=0 Oct 30 18:12:17.175270 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:17.175089 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" Oct 30 18:12:17.175270 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:17.175130 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" event={"ID":"53429703-95ae-4f02-8c26-a9a296c56a05","Type":"ContainerDied","Data":"7e2406130b23b97ceafe24ac4af504bcc85fe97a8a0703755fd7c01cbadefa7e"} Oct 30 18:12:17.175270 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:17.175179 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh" event={"ID":"53429703-95ae-4f02-8c26-a9a296c56a05","Type":"ContainerDied","Data":"cfb7f8588d3dd9cc11e92d7f17b43c471d7e6d8b8686a6c83b57bcbb87a8214b"} Oct 30 18:12:17.175270 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:17.175200 4603 scope.go:122] "RemoveContainer" containerID="7e2406130b23b97ceafe24ac4af504bcc85fe97a8a0703755fd7c01cbadefa7e" Oct 30 18:12:17.185407 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:17.185385 4603 scope.go:122] "RemoveContainer" containerID="7e2406130b23b97ceafe24ac4af504bcc85fe97a8a0703755fd7c01cbadefa7e" Oct 30 18:12:17.185708 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:12:17.185687 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e2406130b23b97ceafe24ac4af504bcc85fe97a8a0703755fd7c01cbadefa7e\": container with ID starting with 7e2406130b23b97ceafe24ac4af504bcc85fe97a8a0703755fd7c01cbadefa7e not found: ID does not exist" containerID="7e2406130b23b97ceafe24ac4af504bcc85fe97a8a0703755fd7c01cbadefa7e" Oct 30 18:12:17.185782 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:17.185715 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e2406130b23b97ceafe24ac4af504bcc85fe97a8a0703755fd7c01cbadefa7e"} err="failed to get container status \"7e2406130b23b97ceafe24ac4af504bcc85fe97a8a0703755fd7c01cbadefa7e\": rpc error: code = NotFound desc = could not find container \"7e2406130b23b97ceafe24ac4af504bcc85fe97a8a0703755fd7c01cbadefa7e\": container with ID starting with 7e2406130b23b97ceafe24ac4af504bcc85fe97a8a0703755fd7c01cbadefa7e not found: ID does not exist" Oct 30 18:12:17.226904 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:17.226882 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh"] Oct 30 18:12:17.238325 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:17.238303 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qmzkh"] Oct 30 18:12:17.410260 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:17.410231 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="049906ec-ccdb-45e9-8fe9-9f50150fc7f6" path="/var/lib/kubelet/pods/049906ec-ccdb-45e9-8fe9-9f50150fc7f6/volumes" Oct 30 18:12:17.410753 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:17.410739 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53429703-95ae-4f02-8c26-a9a296c56a05" path="/var/lib/kubelet/pods/53429703-95ae-4f02-8c26-a9a296c56a05/volumes" Oct 30 18:12:17.411181 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:17.411168 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bf7d070-07da-4ad5-8464-e39192f8e23e" path="/var/lib/kubelet/pods/5bf7d070-07da-4ad5-8464-e39192f8e23e/volumes" Oct 30 18:12:17.676079 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:17.676047 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:19.348144 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:19.347838 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:19.557125 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:19.557094 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 18:12:19.559995 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:19.559973 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 18:12:20.945249 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:20.945213 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:21.932170 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:12:21.932127 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:12:22.491064 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:22.491036 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:12:22.594526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:22.594124 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:24.372525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:24.364704 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:25.076528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.075296 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:25.627529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.607547 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9"] Oct 30 18:12:25.627529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.608477 4603 cpu_manager.go:414] "containerMap: removing container" podUID="5bf7d070-07da-4ad5-8464-e39192f8e23e" containerName="istio-proxy" Oct 30 18:12:25.627529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.608496 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bf7d070-07da-4ad5-8464-e39192f8e23e" containerName="istio-proxy" Oct 30 18:12:25.627529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.608540 4603 cpu_manager.go:414] "containerMap: removing container" podUID="049906ec-ccdb-45e9-8fe9-9f50150fc7f6" containerName="istio-proxy" Oct 30 18:12:25.627529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.608548 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="049906ec-ccdb-45e9-8fe9-9f50150fc7f6" containerName="istio-proxy" Oct 30 18:12:25.627529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.608575 4603 cpu_manager.go:414] "containerMap: removing container" podUID="53429703-95ae-4f02-8c26-a9a296c56a05" containerName="istio-proxy" Oct 30 18:12:25.627529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.608583 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53429703-95ae-4f02-8c26-a9a296c56a05" containerName="istio-proxy" Oct 30 18:12:25.627529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.608753 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bf7d070-07da-4ad5-8464-e39192f8e23e" containerName="istio-proxy" Oct 30 18:12:25.627529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.608766 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="049906ec-ccdb-45e9-8fe9-9f50150fc7f6" containerName="istio-proxy" Oct 30 18:12:25.627529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.608779 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="53429703-95ae-4f02-8c26-a9a296c56a05" containerName="istio-proxy" Oct 30 18:12:25.664564 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.653915 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9"] Oct 30 18:12:25.664564 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.654090 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:25.716532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.715726 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4c8e1a-openshift-default-dockercfg-cm2hk\"" Oct 30 18:12:25.748581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.740688 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-data\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:25.748581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.740801 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-workload-socket\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:25.748581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.740877 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-token\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:25.748581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.740927 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-envoy\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:25.748581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.740965 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fglx\" (UniqueName: \"kubernetes.io/projected/8697996a-d257-4a4e-9c3c-e372ee1b0726-kube-api-access-5fglx\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:25.748581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.741029 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-workload-certs\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:25.748581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.741069 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8697996a-d257-4a4e-9c3c-e372ee1b0726-istiod-ca-cert\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:25.748581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.741153 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-credential-socket\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:25.748581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.741236 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-podinfo\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:25.871537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.844410 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-data\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:25.886850 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.886757 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-workload-socket\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:25.940677 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.940642 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-token\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:25.987557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.979653 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-envoy\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:25.987557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.979725 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fglx\" (UniqueName: \"kubernetes.io/projected/8697996a-d257-4a4e-9c3c-e372ee1b0726-kube-api-access-5fglx\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:25.987557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.979807 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-workload-certs\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:25.987557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.979859 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8697996a-d257-4a4e-9c3c-e372ee1b0726-istiod-ca-cert\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:25.987557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.979965 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-credential-socket\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:25.987557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.980068 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-podinfo\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:25.987557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.849893 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:26.026867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.851107 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-data\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:26.026867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:25.894941 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-workload-socket\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:26.026867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:26.017460 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8697996a-d257-4a4e-9c3c-e372ee1b0726-istiod-ca-cert\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:26.026867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:26.017764 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-workload-certs\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:26.026867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:26.018029 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-credential-socket\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:26.102712 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:26.102675 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-envoy\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:26.201044 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:26.200318 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:26.205281 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:26.205252 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fglx\" (UniqueName: \"kubernetes.io/projected/8697996a-d257-4a4e-9c3c-e372ee1b0726-kube-api-access-5fglx\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:26.229572 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:26.229541 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-podinfo\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:26.260831 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:26.260313 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-token\") pod \"e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:26.393806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:26.393772 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:26.409756 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:26.409132 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:12:26.601955 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:26.592598 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:26.601955 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:26.598145 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:12:26.665535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:26.656644 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:26.920526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:26.904594 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:26.951644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:26.939792 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:27.124557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:27.120737 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:12:27.134131 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:27.134009 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:27.134865 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:27.134837 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:27.258462 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:27.256880 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:12:27.258462 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:27.256946 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" podUID="d5567046-aa24-421a-8339-fdd9cf56e62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:27.370556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:27.359653 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded" Oct 30 18:12:27.446528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:27.402057 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9"] Oct 30 18:12:27.553813 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:27.433398 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:27.553813 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:27.465954 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:27.553813 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:27.549535 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:27.553813 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:27.549601 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:27.683547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:27.606581 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:27.683547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:27.610349 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:27.683547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:27.650387 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:12:27.683547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:27.650452 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:12:27.683547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:27.650494 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:12:27.683547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:27.665282 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:12:27.683547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:27.667701 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:27.683547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:27.670014 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:12:27.912886 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:27.904764 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:27.912886 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:27.905519 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:12:28.119561 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:28.118045 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:12:28.181928 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:28.137114 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:12:28.181928 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:28.154817 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:12:28.181928 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:28.155859 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:12:28.181928 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:28.171339 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": context deadline exceeded" Oct 30 18:12:28.535531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:28.492601 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:28.633894 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:28.585375 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:28.633894 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:28.612280 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:28.633894 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:28.633677 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:28.825959 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:28.825912 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:28.891530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:28.890775 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:12:28.891733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:28.848824 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:12:28.988537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:28.985718 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:29.013532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:29.010637 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:29.165539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:29.161684 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:29.173173 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:29.172362 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:29.206748 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:29.193768 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" event={"ID":"8697996a-d257-4a4e-9c3c-e372ee1b0726","Type":"ContainerStarted","Data":"d558ffa5c3a0949c694fef114919dc5acf05d5fe3a23db7a8352c6fb8157d312"} Oct 30 18:12:29.611526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:29.585681 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:29.635851 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:29.635811 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:12:29.898770 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:29.898671 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:29.969642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:29.969604 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:12:29.969901 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:29.969872 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" podUID="81f49829-73ed-4760-9444-f1793861eba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:29.993175 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:29.993133 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:30.005360 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:30.004579 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:12:30.013704 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:30.013672 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:12:30.166736 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:30.166621 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:12:30.257567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:30.211380 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:12:30.360974 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:30.360931 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:30.543527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:30.535929 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" event={"ID":"8697996a-d257-4a4e-9c3c-e372ee1b0726","Type":"ContainerStarted","Data":"6c609ca59027de3e803458ad36e66f0b143761b51b24b1c73c20c5a00e55b4a1"} Oct 30 18:12:30.615708 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:30.615658 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:30.690528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:30.686589 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:12:30.690528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:30.689610 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" podStartSLOduration=5.689591945 podStartE2EDuration="5.689591945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:30.617556544 +0000 UTC m=+1211.333126418" watchObservedRunningTime="2026-10-30 18:12:30.689591945 +0000 UTC m=+1211.405161816" Oct 30 18:12:30.955627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:30.955522 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:30.996816 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:30.996771 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:31.010266 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:31.010072 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:31.020548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:31.019587 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:31.187543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:31.184685 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:31.211678 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:31.211578 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:31.401431 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:31.401396 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:31.401852 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:31.401828 4603 patch_prober.go:29] interesting pod/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:12:31.402050 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:31.402024 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" podUID="8697996a-d257-4a4e-9c3c-e372ee1b0726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 18:12:31.583404 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:31.583370 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:31.633650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:31.633603 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:31.690901 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:31.690848 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:31.986595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:31.984600 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:32.001542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:31.997583 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:32.036738 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:32.036670 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:12:32.078543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:32.077621 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:32.193594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:32.193462 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:32.216795 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:32.216736 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:32.410899 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:32.410864 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:32.650532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:32.637799 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:12:32.695592 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:32.692394 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:12:32.820691 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:32.820653 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:32.881527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:32.875873 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:12:33.026475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:33.026431 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:33.028401 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:33.028367 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:12:33.039653 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:33.039617 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:12:33.079642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:33.079593 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:33.327852 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:33.327763 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:33.375530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:33.329048 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:33.509261 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:33.508787 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:12:33.509261 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:33.508851 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" podUID="ec3c95aa-215c-4f15-8dc8-bc0beb954236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:33.672696 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:33.672593 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:33.692026 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:33.683883 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:33.693071 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:33.693035 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:33.886180 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:33.886148 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:33.999721 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:33.999624 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:12:34.000002 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:33.999971 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" podUID="b664f4ca-736e-4cb1-991d-985d1e6aff34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:34.064533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:34.054391 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:34.064533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:34.055379 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:34.064533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:34.056245 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:12:34.104829 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:34.104397 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:34.104829 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:34.104629 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:12:34.104829 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:34.104704 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" podUID="4d6cd466-9f15-476d-9f04-6d03176f025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:34.105672 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:34.105569 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:12:34.105672 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:34.105615 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" podUID="187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:34.200530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:34.199811 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:12:34.200530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:34.199875 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" podUID="1d187b51-f5c6-4a63-8300-2f01184a15d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:34.296530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:34.286396 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:12:34.296530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:34.286494 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" podUID="d4a3adf6-fb23-4334-8bab-f800b4e8e089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:34.296530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:34.291671 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:12:34.296530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:34.291721 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" podUID="b6e43478-8cc1-4452-8bc8-fed14f127896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:34.391543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:34.341520 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:34.391959 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:34.391917 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:34.608789 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:34.608671 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:34.685542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:34.684784 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:12:34.707539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:34.706647 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:35.054634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:35.054588 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:35.057197 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:35.055554 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:35.057197 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:35.056397 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:12:35.101150 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:35.101118 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:35.116649 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:35.116606 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:35.393031 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:35.392943 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:35.394748 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:35.393319 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:12:35.619029 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:35.606881 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Oct 30 18:12:35.619029 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:35.608424 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:35.776210 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:35.746609 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:12:35.789541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:35.784552 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:35.893210 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:35.893178 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c"] Oct 30 18:12:35.938446 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:35.938398 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:36.093585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:36.089795 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:36.203346 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:36.090726 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:36.213527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:36.090758 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:36.250529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:36.093448 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c" Oct 30 18:12:36.345590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:36.143769 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:12:36.367536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:36.225234 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5c6p\" (UniqueName: \"kubernetes.io/projected/16ec8bc0-c4ff-443c-91ad-13ad80ff992a-kube-api-access-s5c6p\") pod \"payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c\" (UID: \"16ec8bc0-c4ff-443c-91ad-13ad80ff992a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c" Oct 30 18:12:36.367536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:36.362166 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ec8bc0-c4ff-443c-91ad-13ad80ff992a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c\" (UID: \"16ec8bc0-c4ff-443c-91ad-13ad80ff992a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c" Oct 30 18:12:36.367536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:36.307916 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4c8e1a\"" Oct 30 18:12:36.368649 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:36.308445 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4c8e1a-dockercfg-79666\"" Oct 30 18:12:36.368649 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:36.333559 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c"] Oct 30 18:12:36.368649 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:36.368016 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:12:36.399761 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:36.399046 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4"] Oct 30 18:12:36.421565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:36.416001 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:36.523528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:36.502465 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ec8bc0-c4ff-443c-91ad-13ad80ff992a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c\" (UID: \"16ec8bc0-c4ff-443c-91ad-13ad80ff992a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c" Oct 30 18:12:36.523528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:36.502801 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5c6p\" (UniqueName: \"kubernetes.io/projected/16ec8bc0-c4ff-443c-91ad-13ad80ff992a-kube-api-access-s5c6p\") pod \"payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c\" (UID: \"16ec8bc0-c4ff-443c-91ad-13ad80ff992a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c" Oct 30 18:12:36.523528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:36.504152 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ec8bc0-c4ff-443c-91ad-13ad80ff992a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c\" (UID: \"16ec8bc0-c4ff-443c-91ad-13ad80ff992a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c" Oct 30 18:12:36.654591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:36.626535 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5c6p\" (UniqueName: \"kubernetes.io/projected/16ec8bc0-c4ff-443c-91ad-13ad80ff992a-kube-api-access-s5c6p\") pod \"payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c\" (UID: \"16ec8bc0-c4ff-443c-91ad-13ad80ff992a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c" Oct 30 18:12:36.808585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:36.800814 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:36.824528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:36.811861 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:36.954712 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:36.862247 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4"] Oct 30 18:12:36.960747 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:36.960721 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:12:36.969113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:36.862343 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4" Oct 30 18:12:37.227126 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:37.204594 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:37.305254 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:37.305210 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c" Oct 30 18:12:37.307165 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:37.307140 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:37.322651 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:37.320871 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:37.322651 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:37.321920 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:37.387528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:37.383973 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:37.443201 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:37.391786 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:37.488958 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:37.462890 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:37.687529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:37.660683 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13b740a-1ca9-4c58-a7d2-4beaa132e379-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4\" (UID: \"e13b740a-1ca9-4c58-a7d2-4beaa132e379\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4" Oct 30 18:12:37.688132 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:37.687962 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smbcr\" (UniqueName: \"kubernetes.io/projected/e13b740a-1ca9-4c58-a7d2-4beaa132e379-kube-api-access-smbcr\") pod \"payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4\" (UID: \"e13b740a-1ca9-4c58-a7d2-4beaa132e379\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4" Oct 30 18:12:37.700648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:37.700612 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Oct 30 18:12:37.851542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:37.849336 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:37.871305 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:37.871254 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" podUID="f4343775-794f-4a3c-8732-01eeff138bbe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:37.893017 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:37.880023 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smbcr\" (UniqueName: \"kubernetes.io/projected/e13b740a-1ca9-4c58-a7d2-4beaa132e379-kube-api-access-smbcr\") pod \"payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4\" (UID: \"e13b740a-1ca9-4c58-a7d2-4beaa132e379\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4" Oct 30 18:12:37.893458 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:37.893420 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13b740a-1ca9-4c58-a7d2-4beaa132e379-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4\" (UID: \"e13b740a-1ca9-4c58-a7d2-4beaa132e379\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4" Oct 30 18:12:37.903569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:37.903539 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13b740a-1ca9-4c58-a7d2-4beaa132e379-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4\" (UID: \"e13b740a-1ca9-4c58-a7d2-4beaa132e379\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4" Oct 30 18:12:37.928595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:37.914168 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:37.961199 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:37.961168 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smbcr\" (UniqueName: \"kubernetes.io/projected/e13b740a-1ca9-4c58-a7d2-4beaa132e379-kube-api-access-smbcr\") pod \"payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4\" (UID: \"e13b740a-1ca9-4c58-a7d2-4beaa132e379\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4" Oct 30 18:12:37.978937 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:37.978893 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:12:37.979457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:37.979438 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:12:38.063779 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:38.054974 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:38.179618 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:38.179588 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:12:38.208316 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:38.206638 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:38.451951 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:38.237643 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4" Oct 30 18:12:38.452655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:38.405145 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:12:38.452953 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:38.452926 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:12:38.553848 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:38.553824 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:12:38.561688 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:38.561660 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c"] Oct 30 18:12:38.846613 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:38.846565 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:39.020752 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:38.986164 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4"] Oct 30 18:12:39.107424 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:12:39.097055 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b740a_1ca9_4c58_a7d2_4beaa132e379.slice/crio-738c0d5558801cf9c122f7946ce4dd156e139002fe39ad38fda5930038da3fb9 WatchSource:0}: Error finding container 738c0d5558801cf9c122f7946ce4dd156e139002fe39ad38fda5930038da3fb9: Status 404 returned error can't find the container with id 738c0d5558801cf9c122f7946ce4dd156e139002fe39ad38fda5930038da3fb9 Oct 30 18:12:39.111033 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:39.108103 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c" event={"ID":"16ec8bc0-c4ff-443c-91ad-13ad80ff992a","Type":"ContainerStarted","Data":"8afef7c1bcb66eaee4a6beb0be9b18092e4e6097d4f5f4ba6267b9ae89649a45"} Oct 30 18:12:39.111033 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:39.108171 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c" event={"ID":"16ec8bc0-c4ff-443c-91ad-13ad80ff992a","Type":"ContainerStarted","Data":"32b640e5987f278d92c9a1600e2ab719e735022796cc92c52b34ba8a0980ecac"} Oct 30 18:12:39.111033 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:39.109209 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c" Oct 30 18:12:39.160775 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:39.160721 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c" podStartSLOduration=4.160700684 podStartE2EDuration="4.160700684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:39.158667925 +0000 UTC m=+1219.874237796" watchObservedRunningTime="2026-10-30 18:12:39.160700684 +0000 UTC m=+1219.876270559" Oct 30 18:12:40.120575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:40.120530 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4" event={"ID":"e13b740a-1ca9-4c58-a7d2-4beaa132e379","Type":"ContainerStarted","Data":"8b2c376ba153e389847276f71a1417a80235ea7e1bb5f3119b4cb8cddbabfd2a"} Oct 30 18:12:40.121011 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:40.120595 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4" event={"ID":"e13b740a-1ca9-4c58-a7d2-4beaa132e379","Type":"ContainerStarted","Data":"738c0d5558801cf9c122f7946ce4dd156e139002fe39ad38fda5930038da3fb9"} Oct 30 18:12:40.121011 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:40.120668 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4" Oct 30 18:12:40.161653 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:40.161609 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4" podStartSLOduration=5.161596649 podStartE2EDuration="5.161596649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:40.159749948 +0000 UTC m=+1220.875319823" watchObservedRunningTime="2026-10-30 18:12:40.161596649 +0000 UTC m=+1220.877166524" Oct 30 18:12:40.470216 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:40.470134 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:41.757700 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:41.757659 4603 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:12:41.758094 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:41.757735 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" podUID="02a7d3cd-7df0-41c6-9063-f2caf0e934d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:42.025635 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:42.025597 4603 patch_prober.go:29] interesting pod/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:12:42.025798 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:42.025651 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" podUID="b162d38e-8491-4a60-8e12-c6338131cd03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:42.329015 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:42.328925 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:43.941875 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:43.941840 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:44.152695 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:44.152611 4603 generic.go:356] "Generic (PLEG): container finished" podID="02a7d3cd-7df0-41c6-9063-f2caf0e934d4" containerID="cd9faa3b3a83d54fc93fc5f072721bd4c1d303273fc7835188352f5b849a5827" exitCode=0 Oct 30 18:12:44.152695 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:44.152679 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" event={"ID":"02a7d3cd-7df0-41c6-9063-f2caf0e934d4","Type":"ContainerDied","Data":"cd9faa3b3a83d54fc93fc5f072721bd4c1d303273fc7835188352f5b849a5827"} Oct 30 18:12:44.153089 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:44.153075 4603 scope.go:122] "RemoveContainer" containerID="cd9faa3b3a83d54fc93fc5f072721bd4c1d303273fc7835188352f5b849a5827" Oct 30 18:12:44.154671 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:44.154639 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:44.154745 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:44.154730 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:44.154826 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:44.154809 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:45.160904 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:45.160820 4603 generic.go:356] "Generic (PLEG): container finished" podID="b162d38e-8491-4a60-8e12-c6338131cd03" containerID="9368e601f948256dc178db7455f99ecbd3896b8c9ef80ee1382f6c80eb434483" exitCode=0 Oct 30 18:12:45.160904 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:45.160893 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" event={"ID":"b162d38e-8491-4a60-8e12-c6338131cd03","Type":"ContainerDied","Data":"9368e601f948256dc178db7455f99ecbd3896b8c9ef80ee1382f6c80eb434483"} Oct 30 18:12:45.161416 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:45.161386 4603 scope.go:122] "RemoveContainer" containerID="9368e601f948256dc178db7455f99ecbd3896b8c9ef80ee1382f6c80eb434483" Oct 30 18:12:45.163183 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:45.163151 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:45.163313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:45.163230 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:45.163313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:45.163165 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" event={"ID":"02a7d3cd-7df0-41c6-9063-f2caf0e934d4","Type":"ContainerStarted","Data":"a27e5ec44600057d403fb860e9675144e27c797945be7f4cb1b40c78d5ec00b0"} Oct 30 18:12:45.163313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:45.163273 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:45.740246 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:45.740206 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:45.879528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:45.878976 4603 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:12:45.879528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:45.879045 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" podUID="02a7d3cd-7df0-41c6-9063-f2caf0e934d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 18:12:45.879528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:45.879378 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:12:46.174824 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:46.174784 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" event={"ID":"b162d38e-8491-4a60-8e12-c6338131cd03","Type":"ContainerStarted","Data":"c8dd7267e455e5ee7f017b159860937aaf5fde66bdb3c288958441f62faeb01f"} Oct 30 18:12:46.829530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:46.828910 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:12:46.829530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:46.829039 4603 patch_prober.go:29] interesting pod/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:12:46.829530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:46.829092 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" podUID="b162d38e-8491-4a60-8e12-c6338131cd03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:12:46.875390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:46.875355 4603 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:12:46.875603 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:46.875406 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" podUID="02a7d3cd-7df0-41c6-9063-f2caf0e934d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 18:12:47.456562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:47.454540 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:47.836531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:47.835591 4603 patch_prober.go:29] interesting pod/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:12:47.836531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:47.835656 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" podUID="b162d38e-8491-4a60-8e12-c6338131cd03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:12:47.875463 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:47.875432 4603 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:12:47.875647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:47.875482 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" podUID="02a7d3cd-7df0-41c6-9063-f2caf0e934d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 18:12:48.839727 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:48.839670 4603 patch_prober.go:29] interesting pod/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:12:48.840324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:48.840291 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" podUID="b162d38e-8491-4a60-8e12-c6338131cd03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:12:48.875872 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:48.875842 4603 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:12:48.875995 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:48.875903 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" podUID="02a7d3cd-7df0-41c6-9063-f2caf0e934d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 18:12:49.145584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:49.145378 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:49.299523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:49.295653 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:49.828530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:49.825006 4603 patch_prober.go:29] interesting pod/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:12:49.828530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:49.825076 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" podUID="b162d38e-8491-4a60-8e12-c6338131cd03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:12:49.876047 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:49.876005 4603 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:12:49.877068 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:49.877024 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" podUID="02a7d3cd-7df0-41c6-9063-f2caf0e934d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 18:12:50.833547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:50.832973 4603 patch_prober.go:29] interesting pod/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:12:50.833547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:50.833044 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" podUID="b162d38e-8491-4a60-8e12-c6338131cd03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:12:50.879020 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:50.878976 4603 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:12:50.880558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:50.880488 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" podUID="02a7d3cd-7df0-41c6-9063-f2caf0e934d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 18:12:51.039561 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:51.035108 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:51.136620 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:51.136535 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4" Oct 30 18:12:51.139389 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:51.139367 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c" Oct 30 18:12:51.825421 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:51.825385 4603 patch_prober.go:29] interesting pod/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:12:51.825602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:51.825442 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" podUID="b162d38e-8491-4a60-8e12-c6338131cd03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:12:51.880776 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:51.880743 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:12:51.881298 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:51.881059 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:12:51.881916 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:51.881900 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:12:51.933323 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:12:51.933284 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:12:52.713349 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:52.713317 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:52.859530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:52.859481 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:12:52.860201 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:52.860153 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:12:52.861116 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:52.861093 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:12:54.370134 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:54.370102 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:55.881373 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:55.881340 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:57.458223 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:57.458191 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:59.085187 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:12:59.085154 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:01.043869 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:01.043831 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:02.887627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:02.876027 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:03.837033 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:03.836999 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:04.442147 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:04.442118 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:04.582154 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:04.582119 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:04.846195 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:04.846163 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd"] Oct 30 18:13:04.849386 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:04.849364 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:04.867662 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:04.867631 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0498d2-openshift-default-dockercfg-pzgdv\"" Oct 30 18:13:04.883084 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:04.883058 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd"] Oct 30 18:13:04.930113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:04.930084 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:05.013526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.012813 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-workload-socket\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.013526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.012887 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-token\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.013526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.012991 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-envoy\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.013526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.013061 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-workload-certs\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.013526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.013169 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-credential-socket\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.013526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.013198 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-podinfo\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.013526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.013268 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-data\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.013526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.013298 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrq9p\" (UniqueName: \"kubernetes.io/projected/c7e02b19-7f85-4e69-8a62-f5436875c9e4-kube-api-access-qrq9p\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.013526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.013351 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istiod-ca-cert\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.080639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.077989 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:05.124578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.119526 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-envoy\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.124578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.119700 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-workload-certs\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.124578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.119846 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-credential-socket\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.124578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.119877 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-podinfo\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.124578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.120003 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-data\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.124578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.120030 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrq9p\" (UniqueName: \"kubernetes.io/projected/c7e02b19-7f85-4e69-8a62-f5436875c9e4-kube-api-access-qrq9p\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.124578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.120087 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istiod-ca-cert\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.124578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.120212 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-workload-socket\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.124578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.120282 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-token\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.124578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.121421 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-data\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.152453 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.152414 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-workload-certs\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.152762 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.152739 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-envoy\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.153030 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.153004 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-workload-socket\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.169001 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.168923 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-podinfo\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.169311 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.169290 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-credential-socket\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.170445 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.169995 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istiod-ca-cert\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.189560 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.174747 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-token\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.189833 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.183601 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrq9p\" (UniqueName: \"kubernetes.io/projected/c7e02b19-7f85-4e69-8a62-f5436875c9e4-kube-api-access-qrq9p\") pod \"e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.462055 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.461956 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:05.784761 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.784737 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd"] Oct 30 18:13:05.785168 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:13:05.785145 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e02b19_7f85_4e69_8a62_f5436875c9e4.slice/crio-f2bb89e11d0ebe1ed14efd976412546bf78ec72e1e1c4e05b1d854370eda8576 WatchSource:0}: Error finding container f2bb89e11d0ebe1ed14efd976412546bf78ec72e1e1c4e05b1d854370eda8576: Status 404 returned error can't find the container with id f2bb89e11d0ebe1ed14efd976412546bf78ec72e1e1c4e05b1d854370eda8576 Oct 30 18:13:05.787737 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.787706 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:13:05.787824 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.787767 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:13:05.787824 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:05.787794 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:13:06.190622 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:06.190460 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:06.401555 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:06.401518 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" event={"ID":"c7e02b19-7f85-4e69-8a62-f5436875c9e4","Type":"ContainerStarted","Data":"896109ffdae1339171426490195b8633d98b45ff0736bc0b5614fafbf0653bb3"} Oct 30 18:13:06.401719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:06.401579 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" event={"ID":"c7e02b19-7f85-4e69-8a62-f5436875c9e4","Type":"ContainerStarted","Data":"f2bb89e11d0ebe1ed14efd976412546bf78ec72e1e1c4e05b1d854370eda8576"} Oct 30 18:13:06.461022 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:06.460921 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" podStartSLOduration=2.460905491 podStartE2EDuration="2.460905491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:06.458179329 +0000 UTC m=+1247.173749219" watchObservedRunningTime="2026-10-30 18:13:06.460905491 +0000 UTC m=+1247.176475428" Oct 30 18:13:06.462291 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:06.462265 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:06.463802 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:06.463780 4603 patch_prober.go:29] interesting pod/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:13:06.463898 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:06.463832 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" podUID="c7e02b19-7f85-4e69-8a62-f5436875c9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:13:07.466342 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:07.466313 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:08.089527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:08.085668 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:08.414957 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:08.414883 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:08.415783 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:08.415767 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:13:08.609017 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:08.608964 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:08.884578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:08.884464 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:09.076839 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.076798 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:09.281870 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.281832 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:09.679814 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.679732 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc"] Oct 30 18:13:09.683938 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.683915 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.706234 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.706200 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc"] Oct 30 18:13:09.716104 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.716082 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0bdf03-openshift-default-dockercfg-krrwb\"" Oct 30 18:13:09.769944 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.769920 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:09.838168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.838102 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-token\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.838565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.838537 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-envoy\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.838819 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.838801 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-podinfo\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.838948 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.838934 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istiod-ca-cert\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.839101 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.839073 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-data\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.839293 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.839273 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-credential-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.839481 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.839463 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcvv9\" (UniqueName: \"kubernetes.io/projected/b4eac5a8-12c7-41b6-a8ac-8e570386586c-kube-api-access-wcvv9\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.851665 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.851628 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-workload-certs\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.851937 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.851909 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-workload-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.947460 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.947372 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:09.953334 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.953303 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-token\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.953950 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.953925 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-envoy\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.954271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.954252 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-podinfo\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.954410 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.954395 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istiod-ca-cert\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.954577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.954561 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-data\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.954742 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.954725 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-credential-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.962683 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.962655 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcvv9\" (UniqueName: \"kubernetes.io/projected/b4eac5a8-12c7-41b6-a8ac-8e570386586c-kube-api-access-wcvv9\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.962925 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.962906 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-workload-certs\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.963063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.963049 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-workload-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.986795 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.963676 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-workload-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.987045 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.965348 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istiod-ca-cert\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.987542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.966321 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-data\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.987697 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.966532 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-workload-certs\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.987807 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.966743 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-credential-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.987894 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.978090 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-envoy\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:09.987978 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:09.982089 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-podinfo\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:10.012546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:10.011082 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-token\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:10.029528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:10.028444 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcvv9\" (UniqueName: \"kubernetes.io/projected/b4eac5a8-12c7-41b6-a8ac-8e570386586c-kube-api-access-wcvv9\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:10.118667 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:10.118626 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:10.305839 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:10.305803 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:10.696259 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:10.696234 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc"] Oct 30 18:13:10.698257 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:13:10.698227 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4eac5a8_12c7_41b6_a8ac_8e570386586c.slice/crio-aa91ae0f9cecd80892404bf1b500db60541a38551ff1d8a28e6bef25510ff197 WatchSource:0}: Error finding container aa91ae0f9cecd80892404bf1b500db60541a38551ff1d8a28e6bef25510ff197: Status 404 returned error can't find the container with id aa91ae0f9cecd80892404bf1b500db60541a38551ff1d8a28e6bef25510ff197 Oct 30 18:13:10.701471 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:10.701429 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:13:10.701631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:10.701542 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:13:10.701631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:10.701594 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:13:10.717296 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:10.717268 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:10.880802 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:10.880771 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:11.054532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:11.042569 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm"] Oct 30 18:13:11.054532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:11.046723 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm" Oct 30 18:13:11.077531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:11.076523 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0498d2\"" Oct 30 18:13:11.077531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:11.076707 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0498d2-dockercfg-6bdxv\"" Oct 30 18:13:11.107528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:11.107174 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f158a045-fa10-4b24-ae77-829b4a70b423-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm\" (UID: \"f158a045-fa10-4b24-ae77-829b4a70b423\") " pod="openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm" Oct 30 18:13:11.107528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:11.107272 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj42w\" (UniqueName: \"kubernetes.io/projected/f158a045-fa10-4b24-ae77-829b4a70b423-kube-api-access-jj42w\") pod \"payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm\" (UID: \"f158a045-fa10-4b24-ae77-829b4a70b423\") " pod="openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm" Oct 30 18:13:11.191569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:11.159588 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm"] Oct 30 18:13:11.215529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:11.209750 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f158a045-fa10-4b24-ae77-829b4a70b423-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm\" (UID: \"f158a045-fa10-4b24-ae77-829b4a70b423\") " pod="openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm" Oct 30 18:13:11.215529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:11.209855 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj42w\" (UniqueName: \"kubernetes.io/projected/f158a045-fa10-4b24-ae77-829b4a70b423-kube-api-access-jj42w\") pod \"payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm\" (UID: \"f158a045-fa10-4b24-ae77-829b4a70b423\") " pod="openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm" Oct 30 18:13:11.230593 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:11.229609 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f158a045-fa10-4b24-ae77-829b4a70b423-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm\" (UID: \"f158a045-fa10-4b24-ae77-829b4a70b423\") " pod="openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm" Oct 30 18:13:11.299290 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:11.299217 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj42w\" (UniqueName: \"kubernetes.io/projected/f158a045-fa10-4b24-ae77-829b4a70b423-kube-api-access-jj42w\") pod \"payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm\" (UID: \"f158a045-fa10-4b24-ae77-829b4a70b423\") " pod="openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm" Oct 30 18:13:11.436873 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:11.436786 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" event={"ID":"b4eac5a8-12c7-41b6-a8ac-8e570386586c","Type":"ContainerStarted","Data":"33046b182173b53bab6743bc3bfae8d29b429cb5333ea2642320c68b70a438cc"} Oct 30 18:13:11.436873 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:11.436836 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" event={"ID":"b4eac5a8-12c7-41b6-a8ac-8e570386586c","Type":"ContainerStarted","Data":"aa91ae0f9cecd80892404bf1b500db60541a38551ff1d8a28e6bef25510ff197"} Oct 30 18:13:11.481136 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:11.481091 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" podStartSLOduration=2.481077249 podStartE2EDuration="2.481077249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:11.480185898 +0000 UTC m=+1252.195755783" watchObservedRunningTime="2026-10-30 18:13:11.481077249 +0000 UTC m=+1252.196647123" Oct 30 18:13:11.547466 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:11.547432 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm" Oct 30 18:13:11.972733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:11.966184 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:12.215527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:12.210349 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm"] Oct 30 18:13:12.312806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:12.312780 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:12.322615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:12.322591 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:12.444304 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:12.444265 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm" event={"ID":"f158a045-fa10-4b24-ae77-829b4a70b423","Type":"ContainerStarted","Data":"1ccef2a829da1f10efb8bd90ef4cbf7ca4e4c32c011d2e750bd0b5de3d2f09a5"} Oct 30 18:13:12.444434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:12.444323 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm" event={"ID":"f158a045-fa10-4b24-ae77-829b4a70b423","Type":"ContainerStarted","Data":"84e83ead594f41411ea063882b90b8fe5860de19a336981d3f819b11d3bd7af2"} Oct 30 18:13:12.444434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:12.444354 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm" Oct 30 18:13:12.444642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:12.444625 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:12.445614 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:12.445592 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:13:12.485003 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:12.484964 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm" podStartSLOduration=2.484952882 podStartE2EDuration="2.484952882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:12.483421155 +0000 UTC m=+1253.198991029" watchObservedRunningTime="2026-10-30 18:13:12.484952882 +0000 UTC m=+1253.200522757" Oct 30 18:13:13.574602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:13.574565 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:13.731819 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:13.731782 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:14.199062 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:14.199032 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:15.263124 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:15.263023 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:16.848298 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:16.848260 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:17.333882 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:17.333847 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:18.239027 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:18.238967 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:18.438375 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:18.438344 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:20.229196 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:20.229140 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:21.833141 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:21.833097 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:21.992317 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:13:21.992268 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:13:23.454279 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:23.454250 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm" Oct 30 18:13:23.517728 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:23.517697 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:23.695474 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:23.695427 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:23.874996 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:23.874965 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:24.016282 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:24.016250 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:25.358749 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:25.358715 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:25.598794 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:25.598760 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:25.749275 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:25.749202 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf"] Oct 30 18:13:25.753474 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:25.753452 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf" Oct 30 18:13:25.754088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:25.754065 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:25.774983 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:25.774964 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf"] Oct 30 18:13:25.781538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:25.781479 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0bdf03\"" Oct 30 18:13:25.781538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:25.781490 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0bdf03-dockercfg-tgcdc\"" Oct 30 18:13:25.845568 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:25.845540 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5xzs\" (UniqueName: \"kubernetes.io/projected/44c65786-428f-4ac0-97db-020b7a909823-kube-api-access-t5xzs\") pod \"payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf\" (UID: \"44c65786-428f-4ac0-97db-020b7a909823\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf" Oct 30 18:13:25.845738 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:25.845718 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44c65786-428f-4ac0-97db-020b7a909823-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf\" (UID: \"44c65786-428f-4ac0-97db-020b7a909823\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf" Oct 30 18:13:25.963530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:25.952613 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5xzs\" (UniqueName: \"kubernetes.io/projected/44c65786-428f-4ac0-97db-020b7a909823-kube-api-access-t5xzs\") pod \"payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf\" (UID: \"44c65786-428f-4ac0-97db-020b7a909823\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf" Oct 30 18:13:25.993531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:25.964006 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44c65786-428f-4ac0-97db-020b7a909823-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf\" (UID: \"44c65786-428f-4ac0-97db-020b7a909823\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf" Oct 30 18:13:26.001573 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:25.971975 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44c65786-428f-4ac0-97db-020b7a909823-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf\" (UID: \"44c65786-428f-4ac0-97db-020b7a909823\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf" Oct 30 18:13:26.117530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:26.096236 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5xzs\" (UniqueName: \"kubernetes.io/projected/44c65786-428f-4ac0-97db-020b7a909823-kube-api-access-t5xzs\") pod \"payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf\" (UID: \"44c65786-428f-4ac0-97db-020b7a909823\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf" Oct 30 18:13:26.368836 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:26.368799 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf" Oct 30 18:13:26.696105 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:26.696077 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf"] Oct 30 18:13:26.696857 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:13:26.696835 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c65786_428f_4ac0_97db_020b7a909823.slice/crio-c093e8cb062e18999c0dd9f5a8b649cbba5270326a1348f4ec0c0ede8e520021 WatchSource:0}: Error finding container c093e8cb062e18999c0dd9f5a8b649cbba5270326a1348f4ec0c0ede8e520021: Status 404 returned error can't find the container with id c093e8cb062e18999c0dd9f5a8b649cbba5270326a1348f4ec0c0ede8e520021 Oct 30 18:13:27.146443 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:27.146410 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:27.572544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:27.572488 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf" event={"ID":"44c65786-428f-4ac0-97db-020b7a909823","Type":"ContainerStarted","Data":"ca53c7a8674652eebfcd0e311f991ac7dae71ae81840e298a562d686c96b3d6f"} Oct 30 18:13:27.572998 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:27.572568 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf" event={"ID":"44c65786-428f-4ac0-97db-020b7a909823","Type":"ContainerStarted","Data":"c093e8cb062e18999c0dd9f5a8b649cbba5270326a1348f4ec0c0ede8e520021"} Oct 30 18:13:27.572998 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:27.572597 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf" Oct 30 18:13:27.613372 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:27.613316 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf" podStartSLOduration=2.6132992489999998 podStartE2EDuration="2.613299249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:27.612212929 +0000 UTC m=+1268.327782826" watchObservedRunningTime="2026-10-30 18:13:27.613299249 +0000 UTC m=+1268.328869124" Oct 30 18:13:28.852913 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:28.852879 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:30.435076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:30.435035 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:30.597952 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:30.597918 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:13:32.239343 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:32.239292 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:33.992544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:33.991605 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:34.003530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:34.002597 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:34.023629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:34.010583 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:34.023629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:34.016580 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:13:34.178541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:34.175820 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:34.205543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:34.202631 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:34.639117 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:34.639071 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:34.698528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:34.695754 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:34.738528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:34.737586 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:35.014275 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:35.011781 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:35.014275 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:35.012260 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:13:35.014275 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:35.013060 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:13:35.014275 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:35.013573 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:13:35.018551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:35.015388 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:35.039534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:35.018759 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:35.039534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:35.019142 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:13:35.061527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:35.052842 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:35.061527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:35.053448 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:13:35.192646 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:35.192598 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:35.215537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:35.199781 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:13:35.242531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:35.215836 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:13:35.870553 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:35.858630 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:13:35.870553 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:35.522678 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:35.870553 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:35.744127 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:35.941544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:35.663536 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:35.941544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:35.898545 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:13:35.941544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:35.898631 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:13:36.019544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:36.019445 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:36.050557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:36.040398 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:36.079543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:36.067524 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:13:36.110031 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:36.109626 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:13:36.326073 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:36.257423 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:13:36.917171 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:36.917123 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:13:37.000893 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:37.000844 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:37.048373 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:37.009144 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:37.093529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:37.021699 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:13:37.108625 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:37.107836 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:37.123938 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:37.123685 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:37.163707 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:37.126796 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:37.263744 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:37.263610 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:13:37.271618 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:37.271570 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:37.367538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:37.353777 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:37.922695 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:37.922649 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:37.995872 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:37.995672 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:37.995872 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:37.995802 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" podUID="d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.084594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:38.047627 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:13:38.135530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:38.125655 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:38.135530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:38.127602 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" podUID="f7882513-ef6d-4b11-871e-39a29ad022df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.135530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:38.126837 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:13:38.135530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:38.125972 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:13:38.176585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:38.126880 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:13:38.176585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:38.126913 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.209241 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:38.178156 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.263954 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:38.263909 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.282851 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:38.282810 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:13:38.450525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:38.426080 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:38.450525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:38.426145 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" podUID="58e2469b-dc0d-40ab-bd74-a47c4a9e438b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.479394 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:38.479350 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.630639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:38.630590 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.662570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:38.658843 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf" Oct 30 18:13:38.950659 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:38.950614 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:39.129697 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:39.129642 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:13:39.155995 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:39.140629 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:13:39.156910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:39.140669 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:13:39.157371 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:39.140691 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:13:39.157811 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:39.142071 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:13:39.279592 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:39.271058 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:39.292684 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:39.292641 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:39.573610 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:39.573190 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:39.930066 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:39.929957 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:39.930368 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:39.930336 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" podUID="47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:39.972093 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:39.972051 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:13:40.158668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:40.158610 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:13:40.160415 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:40.159267 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.160415 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:40.159302 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.160415 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:40.159347 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.166696 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:40.166013 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.272726 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:40.272678 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.273750 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:40.273633 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:40.273750 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:40.273682 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" podUID="db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.309646 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:40.309597 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.311243 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:40.311206 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.985530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:40.978613 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:41.177666 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:41.174784 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:41.177666 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:41.175180 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:41.177666 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:41.175439 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:13:41.177666 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:41.175718 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:13:41.190543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:41.178625 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:13:41.297526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:41.280623 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:13:41.324835 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:41.324792 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:13:41.602716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:41.602611 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:41.603030 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:41.603000 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" podUID="c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:41.988781 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:41.988666 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:42.179053 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:42.177323 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:42.184528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:42.182989 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:42.215703 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:42.215659 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:42.250590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:42.231654 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:13:42.353539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:42.253999 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:42.353539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:42.269136 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:42.353539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:42.298901 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:42.353881 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:42.353562 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" podUID="d5567046-aa24-421a-8339-fdd9cf56e62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:42.353881 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:42.326223 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:42.357529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:42.298973 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:42.998642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:42.998597 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:13:43.189448 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:43.189402 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:13:43.191172 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:43.191140 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:43.225875 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:43.225815 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:13:43.339539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:43.336934 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:13:43.360540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:43.351605 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:43.360540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:43.351666 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" podUID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:43.360540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:43.355571 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:13:43.360540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:43.355927 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:13:43.360540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:43.357010 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:13:43.732603 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:43.720542 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:44.000583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:44.000465 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:44.219524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:44.192965 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:44.253213 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:44.199687 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:13:44.253213 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:44.247664 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:44.338681 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:44.338580 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:44.359539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:44.357669 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:13:44.359539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:44.358025 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:13:44.359539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:44.359270 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:13:44.598270 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:44.598222 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:44.665934 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:44.664889 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:13:44.749773 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:44.749744 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:13:44.753980 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:44.753960 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:13:44.779328 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:44.779306 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:13:44.780874 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:44.780852 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:13:44.781823 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:44.781800 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:13:44.781927 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:44.781856 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:13:44.784013 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:44.783992 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:13:45.414654 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:45.414624 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:45.578301 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:45.578264 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:45.734236 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:45.734148 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:47.345489 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:47.345455 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:48.857704 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:48.857643 4603 patch_prober.go:29] interesting pod/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:48.858794 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:48.858759 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" podUID="8697996a-d257-4a4e-9c3c-e372ee1b0726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:49.117156 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:49.117068 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:49.928913 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:49.928864 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:50.068776 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:50.068736 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:50.943579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:50.943542 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:51.930362 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:13:51.930273 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:13:52.293034 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:52.293007 4603 generic.go:356] "Generic (PLEG): container finished" podID="8697996a-d257-4a4e-9c3c-e372ee1b0726" containerID="6c609ca59027de3e803458ad36e66f0b143761b51b24b1c73c20c5a00e55b4a1" exitCode=0 Oct 30 18:13:52.293451 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:52.293080 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" event={"ID":"8697996a-d257-4a4e-9c3c-e372ee1b0726","Type":"ContainerDied","Data":"6c609ca59027de3e803458ad36e66f0b143761b51b24b1c73c20c5a00e55b4a1"} Oct 30 18:13:52.293494 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:52.293457 4603 scope.go:122] "RemoveContainer" containerID="6c609ca59027de3e803458ad36e66f0b143761b51b24b1c73c20c5a00e55b4a1" Oct 30 18:13:52.294934 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:52.294904 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:13:52.295046 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:52.294990 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:13:52.295111 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:52.295041 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:13:52.542445 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:52.542408 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:53.328731 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:53.328700 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" event={"ID":"8697996a-d257-4a4e-9c3c-e372ee1b0726","Type":"ContainerStarted","Data":"69ebf57a80b52c2f71efea3baa0180f3d7ffea8756fe47c4802e0999d3e17183"} Oct 30 18:13:53.408526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:53.405619 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:13:53.410639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:53.410609 4603 patch_prober.go:29] interesting pod/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:13:53.410894 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:53.410857 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" podUID="8697996a-d257-4a4e-9c3c-e372ee1b0726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 18:13:54.243571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:54.243498 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:54.401564 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:54.400794 4603 patch_prober.go:29] interesting pod/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:13:54.401564 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:54.400874 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" podUID="8697996a-d257-4a4e-9c3c-e372ee1b0726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 18:13:54.666110 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:54.666075 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:55.412530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:55.409376 4603 patch_prober.go:29] interesting pod/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:13:55.415486 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:55.415407 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" podUID="8697996a-d257-4a4e-9c3c-e372ee1b0726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 18:13:56.138987 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:56.138929 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:56.399587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:56.397844 4603 patch_prober.go:29] interesting pod/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:13:56.399587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:56.397911 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" podUID="8697996a-d257-4a4e-9c3c-e372ee1b0726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 18:13:56.974324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:56.974296 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:57.169953 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:57.169918 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:57.409528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:57.399814 4603 patch_prober.go:29] interesting pod/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:13:57.409528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:57.399868 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" podUID="8697996a-d257-4a4e-9c3c-e372ee1b0726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 18:13:58.134253 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:58.127245 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:58.395252 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:58.395162 4603 patch_prober.go:29] interesting pod/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:13:58.395535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:58.395478 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" podUID="8697996a-d257-4a4e-9c3c-e372ee1b0726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 18:13:59.398785 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:59.398754 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:13:59.399255 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:59.399143 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:13:59.399928 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:59.399907 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:13:59.929714 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:13:59.929596 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:01.600523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:01.598012 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:03.243929 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:03.243852 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:04.844201 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:04.844144 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:06.624415 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:06.624383 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:07.562610 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:07.562571 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:07.730527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:07.726187 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:07.871892 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:07.871817 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:08.103536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:08.099759 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:08.448168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:08.448135 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:09.662971 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:09.662935 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:10.128413 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:10.128368 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:11.885443 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:11.885409 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:13.559470 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:13.559436 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:14.312308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:14.312282 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:15.153746 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:15.153710 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:16.843124 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:16.843087 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:18.445035 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:18.445005 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:20.075525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:20.066786 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:21.487266 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:21.487235 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4"] Oct 30 18:14:21.527765 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:21.527735 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c"] Oct 30 18:14:21.649612 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:21.649582 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:21.801094 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:21.801057 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:21.936813 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:14:21.936779 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:14:23.316013 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:23.315967 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:25.028996 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:25.028960 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:26.488545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:26.488481 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4" podUID="e13b740a-1ca9-4c58-a7d2-4beaa132e379" containerName="payload-processing" containerID="cri-o://8b2c376ba153e389847276f71a1417a80235ea7e1bb5f3119b4cb8cddbabfd2a" gracePeriod=25 Oct 30 18:14:26.528817 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:26.528788 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c" podUID="16ec8bc0-c4ff-443c-91ad-13ad80ff992a" containerName="payload-pre-processing" containerID="cri-o://8afef7c1bcb66eaee4a6beb0be9b18092e4e6097d4f5f4ba6267b9ae89649a45" gracePeriod=25 Oct 30 18:14:26.632599 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:26.632570 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:26.740781 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:26.740548 4603 generic.go:356] "Generic (PLEG): container finished" podID="16ec8bc0-c4ff-443c-91ad-13ad80ff992a" containerID="8afef7c1bcb66eaee4a6beb0be9b18092e4e6097d4f5f4ba6267b9ae89649a45" exitCode=0 Oct 30 18:14:26.740781 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:26.740654 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c" event={"ID":"16ec8bc0-c4ff-443c-91ad-13ad80ff992a","Type":"ContainerDied","Data":"8afef7c1bcb66eaee4a6beb0be9b18092e4e6097d4f5f4ba6267b9ae89649a45"} Oct 30 18:14:26.743385 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:26.743149 4603 generic.go:356] "Generic (PLEG): container finished" podID="e13b740a-1ca9-4c58-a7d2-4beaa132e379" containerID="8b2c376ba153e389847276f71a1417a80235ea7e1bb5f3119b4cb8cddbabfd2a" exitCode=0 Oct 30 18:14:26.743385 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:26.743319 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4" event={"ID":"e13b740a-1ca9-4c58-a7d2-4beaa132e379","Type":"ContainerDied","Data":"8b2c376ba153e389847276f71a1417a80235ea7e1bb5f3119b4cb8cddbabfd2a"} Oct 30 18:14:26.928138 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:26.928110 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4" Oct 30 18:14:26.955118 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:26.955095 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c" Oct 30 18:14:27.016611 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:27.016532 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16ec8bc0-c4ff-443c-91ad-13ad80ff992a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ec8bc0-c4ff-443c-91ad-13ad80ff992a-plugins-config-volume\") pod \"16ec8bc0-c4ff-443c-91ad-13ad80ff992a\" (UID: \"16ec8bc0-c4ff-443c-91ad-13ad80ff992a\") " Oct 30 18:14:27.016799 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:27.016628 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16ec8bc0-c4ff-443c-91ad-13ad80ff992a-kube-api-access-s5c6p\" (UniqueName: \"kubernetes.io/projected/16ec8bc0-c4ff-443c-91ad-13ad80ff992a-kube-api-access-s5c6p\") pod \"16ec8bc0-c4ff-443c-91ad-13ad80ff992a\" (UID: \"16ec8bc0-c4ff-443c-91ad-13ad80ff992a\") " Oct 30 18:14:27.016799 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:27.016693 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e13b740a-1ca9-4c58-a7d2-4beaa132e379-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13b740a-1ca9-4c58-a7d2-4beaa132e379-plugins-config-volume\") pod \"e13b740a-1ca9-4c58-a7d2-4beaa132e379\" (UID: \"e13b740a-1ca9-4c58-a7d2-4beaa132e379\") " Oct 30 18:14:27.016936 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:27.016884 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13b740a-1ca9-4c58-a7d2-4beaa132e379-kube-api-access-smbcr\" (UniqueName: \"kubernetes.io/projected/e13b740a-1ca9-4c58-a7d2-4beaa132e379-kube-api-access-smbcr\") pod \"e13b740a-1ca9-4c58-a7d2-4beaa132e379\" (UID: \"e13b740a-1ca9-4c58-a7d2-4beaa132e379\") " Oct 30 18:14:27.016992 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:27.016953 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ec8bc0-c4ff-443c-91ad-13ad80ff992a-plugins-config-volume" pod "16ec8bc0-c4ff-443c-91ad-13ad80ff992a" (UID: "16ec8bc0-c4ff-443c-91ad-13ad80ff992a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:27.017749 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:27.017436 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13b740a-1ca9-4c58-a7d2-4beaa132e379-plugins-config-volume" pod "e13b740a-1ca9-4c58-a7d2-4beaa132e379" (UID: "e13b740a-1ca9-4c58-a7d2-4beaa132e379"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:27.017749 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:27.017689 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ec8bc0-c4ff-443c-91ad-13ad80ff992a-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:14:27.017749 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:27.017710 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13b740a-1ca9-4c58-a7d2-4beaa132e379-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:14:27.019646 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:27.019586 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13b740a-1ca9-4c58-a7d2-4beaa132e379-kube-api-access-smbcr" pod "e13b740a-1ca9-4c58-a7d2-4beaa132e379" (UID: "e13b740a-1ca9-4c58-a7d2-4beaa132e379"). InnerVolumeSpecName "kube-api-access-smbcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:27.019860 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:27.019833 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ec8bc0-c4ff-443c-91ad-13ad80ff992a-kube-api-access-s5c6p" pod "16ec8bc0-c4ff-443c-91ad-13ad80ff992a" (UID: "16ec8bc0-c4ff-443c-91ad-13ad80ff992a"). InnerVolumeSpecName "kube-api-access-s5c6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:27.118699 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:27.118670 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5c6p\" (UniqueName: \"kubernetes.io/projected/16ec8bc0-c4ff-443c-91ad-13ad80ff992a-kube-api-access-s5c6p\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:14:27.118699 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:27.118693 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smbcr\" (UniqueName: \"kubernetes.io/projected/e13b740a-1ca9-4c58-a7d2-4beaa132e379-kube-api-access-smbcr\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:14:27.750331 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:27.750251 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c" Oct 30 18:14:27.750331 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:27.750259 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c" event={"ID":"16ec8bc0-c4ff-443c-91ad-13ad80ff992a","Type":"ContainerDied","Data":"32b640e5987f278d92c9a1600e2ab719e735022796cc92c52b34ba8a0980ecac"} Oct 30 18:14:27.750331 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:27.750312 4603 scope.go:122] "RemoveContainer" containerID="8afef7c1bcb66eaee4a6beb0be9b18092e4e6097d4f5f4ba6267b9ae89649a45" Oct 30 18:14:27.751884 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:27.751866 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4" Oct 30 18:14:27.751996 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:27.751911 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4" event={"ID":"e13b740a-1ca9-4c58-a7d2-4beaa132e379","Type":"ContainerDied","Data":"738c0d5558801cf9c122f7946ce4dd156e139002fe39ad38fda5930038da3fb9"} Oct 30 18:14:27.761334 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:27.761318 4603 scope.go:122] "RemoveContainer" containerID="8b2c376ba153e389847276f71a1417a80235ea7e1bb5f3119b4cb8cddbabfd2a" Oct 30 18:14:27.805119 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:27.805096 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4"] Oct 30 18:14:27.815800 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:27.815780 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c8e1a-69c984459d-jwnj4"] Oct 30 18:14:27.839205 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:27.839183 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c"] Oct 30 18:14:27.843442 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:27.843424 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c8e1a-6b64f8b84c-d4b7c"] Oct 30 18:14:28.340408 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:28.340374 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:29.414539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:29.412399 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16ec8bc0-c4ff-443c-91ad-13ad80ff992a" path="/var/lib/kubelet/pods/16ec8bc0-c4ff-443c-91ad-13ad80ff992a/volumes" Oct 30 18:14:29.414539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:29.413201 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e13b740a-1ca9-4c58-a7d2-4beaa132e379" path="/var/lib/kubelet/pods/e13b740a-1ca9-4c58-a7d2-4beaa132e379/volumes" Oct 30 18:14:30.027842 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:30.027807 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:31.623844 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:31.623792 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:33.262362 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:33.262322 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:34.606261 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:34.606233 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:35.122531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:35.120885 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:35.269312 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:35.269270 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:35.645363 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:35.645333 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:14:35.778143 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:35.778103 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:37.157883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:37.157853 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:38.927524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:38.925211 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:39.041534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:39.031616 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:39.091659 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:39.091615 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:39.135717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:39.135671 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:14:39.136676 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:39.136037 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:14:39.169529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:39.158152 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:39.294525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:39.279629 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:39.298525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:39.298468 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:39.470661 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:39.470625 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:39.761733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:39.761644 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:39.925927 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:39.925881 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:39.926678 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:39.926658 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:14:40.330617 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:40.296306 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:40.330617 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:40.298482 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:14:40.330617 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:40.299257 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:40.330617 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:40.299299 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:40.330617 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:40.300464 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:14:40.330617 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:40.300536 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:14:40.330617 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:40.302215 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:14:40.330617 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:40.314915 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:14:40.330617 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:40.316318 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:14:40.330617 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:40.316638 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:14:40.335522 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:40.332437 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:40.335522 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:40.332818 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:14:40.335522 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:40.334033 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:14:40.335522 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:40.334311 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:14:40.362077 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:40.362043 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Oct 30 18:14:40.974749 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:40.974695 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:14:41.303537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:41.299426 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:14:41.310423 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:41.310380 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:14:41.311698 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:41.311669 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:14:41.313202 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:41.313180 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:14:41.330747 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:41.330712 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:41.344439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:41.333419 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:41.344439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:41.334114 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:41.615875 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:41.615768 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:41.880576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:41.816642 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" podUID="c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:41.984686 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:41.984641 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:14:42.352530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:42.338662 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:14:42.368067 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:42.368016 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:42.388649 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:42.388594 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:42.527176 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:42.474560 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:14:42.527176 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:42.476525 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:42.527176 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:42.476600 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" podUID="d5567046-aa24-421a-8339-fdd9cf56e62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:42.527176 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:42.500024 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:14:42.527176 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:42.500416 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:14:42.527176 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:42.500741 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:14:42.837718 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:42.808407 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:43.016537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:43.014636 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:43.061411 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:43.058151 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:43.242275 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:43.242184 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl"] Oct 30 18:14:43.242772 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:43.242739 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" podUID="b162d38e-8491-4a60-8e12-c6338131cd03" containerName="istio-proxy" containerID="cri-o://c8dd7267e455e5ee7f017b159860937aaf5fde66bdb3c288958441f62faeb01f" gracePeriod=30 Oct 30 18:14:43.371637 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:43.371592 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:43.390675 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:43.390637 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:43.394528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:43.390932 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" podUID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:43.424021 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:43.400914 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:43.513630 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:43.428582 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:43.513630 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:43.491622 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:43.513630 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:43.502588 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:43.513630 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:43.502918 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:14:43.513630 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:43.503161 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:14:43.521640 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:43.521562 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:44.024019 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:44.022095 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:44.424762 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:44.424626 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:14:44.513806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:44.443554 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:14:44.530529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:44.514847 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:14:44.556528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:44.516551 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:44.581917 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:44.516588 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:14:44.585605 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:44.516667 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:14:44.605528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:44.531629 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:44.622643 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:44.622594 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:44.918375 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:44.918312 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:44.918766 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:44.918733 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" podUID="81f49829-73ed-4760-9444-f1793861eba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:45.023686 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:45.023627 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:14:45.519644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:45.519598 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:14:45.520595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:45.520559 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:14:45.521682 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:45.521653 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:45.549528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:45.546817 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:45.583873 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:45.583824 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:45.585660 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:45.585629 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:14:45.605680 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:45.605634 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:14:45.607825 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:45.607333 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:45.884599 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:45.867609 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:46.046549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:46.044832 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:14:46.522655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:46.522608 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:14:46.523718 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:46.523248 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:14:46.552772 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:46.552727 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:46.608530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:46.591594 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:46.608530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:46.592001 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:14:46.609819 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:46.609780 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:46.610336 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:46.610308 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:14:47.062927 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:47.062878 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:47.139074 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:47.139011 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Oct 30 18:14:47.364723 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:47.364614 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:47.596528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:47.567643 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:14:47.596528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:47.568110 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:47.596528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:47.569565 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:47.618015 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:47.597550 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:47.618015 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:47.597989 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:14:47.720551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:47.710771 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:47.720551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:47.711157 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:14:47.826454 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:47.826417 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:48.144969 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:48.130673 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": context deadline exceeded" Oct 30 18:14:48.220916 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:48.220843 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:14:48.320922 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:48.318682 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:48.447640 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:48.426234 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:48.550566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:48.538649 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:48.550566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:48.538720 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" podUID="ec3c95aa-215c-4f15-8dc8-bc0beb954236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:48.605550 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:48.588023 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:14:48.605550 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:48.588422 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:14:48.605550 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:48.588747 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:14:48.605550 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:48.597708 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:48.605550 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:48.598165 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:48.613962 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:48.613899 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:48.735589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:48.735465 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:14:48.736180 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:48.736146 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:14:48.813970 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:48.813931 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2"] Oct 30 18:14:48.845529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:48.841934 4603 cpu_manager.go:414] "containerMap: removing container" podUID="e13b740a-1ca9-4c58-a7d2-4beaa132e379" containerName="payload-processing" Oct 30 18:14:48.878702 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:48.878662 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13b740a-1ca9-4c58-a7d2-4beaa132e379" containerName="payload-processing" Oct 30 18:14:48.909532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:48.895641 4603 cpu_manager.go:414] "containerMap: removing container" podUID="16ec8bc0-c4ff-443c-91ad-13ad80ff992a" containerName="payload-pre-processing" Oct 30 18:14:48.909787 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:48.909760 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16ec8bc0-c4ff-443c-91ad-13ad80ff992a" containerName="payload-pre-processing" Oct 30 18:14:48.923533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:48.910182 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="16ec8bc0-c4ff-443c-91ad-13ad80ff992a" containerName="payload-pre-processing" Oct 30 18:14:49.017636 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.017535 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13b740a-1ca9-4c58-a7d2-4beaa132e379" containerName="payload-processing" Oct 30 18:14:49.027680 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:48.919228 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:49.094566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.078568 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:49.136547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.117641 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:49.136547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.117712 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" podUID="b664f4ca-736e-4cb1-991d-985d1e6aff34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:49.136547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.118403 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c8f010-openshift-default-dockercfg-l259f\"" Oct 30 18:14:49.140540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.140488 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2"] Oct 30 18:14:49.207642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.205934 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-data\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:49.207642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.206078 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-credential-socket\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:49.207642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.206151 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctdkv\" (UniqueName: \"kubernetes.io/projected/3bb952b3-3096-442b-8ab6-2909cee2c021-kube-api-access-ctdkv\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:49.207642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.206232 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-workload-certs\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:49.207642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.206353 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-podinfo\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:49.207642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.206406 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-token\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:49.207642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.206531 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb952b3-3096-442b-8ab6-2909cee2c021-istiod-ca-cert\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:49.207642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.206639 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-envoy\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:49.207642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.206743 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-workload-socket\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:49.243858 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.243803 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:49.264394 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.264332 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:49.286696 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.286652 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" podUID="1d187b51-f5c6-4a63-8300-2f01184a15d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:49.325528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.321490 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-workload-certs\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:49.776550 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.340148 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-workload-certs\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:49.777064 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.353362 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:49.777261 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.372708 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:49.777466 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.598604 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:49.798711 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.598646 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:49.798711 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.598676 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:49.798711 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.598697 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:49.798711 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.598718 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:14:49.798711 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.774264 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:49.798711 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.774303 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:14:49.825299 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.436633 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:49.825600 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.825565 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" podUID="d4a3adf6-fb23-4334-8bab-f800b4e8e089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:49.830526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.826890 4603 generic.go:356] "Generic (PLEG): container finished" podID="b162d38e-8491-4a60-8e12-c6338131cd03" containerID="c8dd7267e455e5ee7f017b159860937aaf5fde66bdb3c288958441f62faeb01f" exitCode=0 Oct 30 18:14:49.909535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.897747 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" event={"ID":"b162d38e-8491-4a60-8e12-c6338131cd03","Type":"ContainerDied","Data":"c8dd7267e455e5ee7f017b159860937aaf5fde66bdb3c288958441f62faeb01f"} Oct 30 18:14:49.909910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.909884 4603 scope.go:122] "RemoveContainer" containerID="9368e601f948256dc178db7455f99ecbd3896b8c9ef80ee1382f6c80eb434483" Oct 30 18:14:49.952327 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:49.944656 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" podUID="187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:50.007668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.001045 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-podinfo\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:50.007668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.001149 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-token\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:50.007668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.001291 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb952b3-3096-442b-8ab6-2909cee2c021-istiod-ca-cert\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:50.007668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.001396 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-envoy\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:50.007668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.001569 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-workload-socket\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:50.007668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.001657 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-data\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:50.007668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.001732 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-credential-socket\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:50.007668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.001817 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctdkv\" (UniqueName: \"kubernetes.io/projected/3bb952b3-3096-442b-8ab6-2909cee2c021-kube-api-access-ctdkv\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:50.071585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.021713 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-credential-socket\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:50.084727 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.079957 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-data\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:50.084971 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.064493 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-workload-socket\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:50.088534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.086996 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb952b3-3096-442b-8ab6-2909cee2c021-istiod-ca-cert\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:50.101527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.098228 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctdkv\" (UniqueName: \"kubernetes.io/projected/3bb952b3-3096-442b-8ab6-2909cee2c021-kube-api-access-ctdkv\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:50.121558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.114589 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-token\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:50.244425 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.176054 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-envoy\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:50.291708 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.287621 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:50.329311 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.327684 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Oct 30 18:14:50.422171 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.422136 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-podinfo\") pod \"e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:50.490438 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.490350 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:50.586301 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.585611 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:14:50.692562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.685210 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:50.784641 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.784594 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:50.856994 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.850836 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:50.865733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.865688 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:50.873622 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.873579 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:14:50.892533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.880998 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-credential-socket\") pod \"b162d38e-8491-4a60-8e12-c6338131cd03\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " Oct 30 18:14:50.897085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.897046 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-istio-envoy\") pod \"b162d38e-8491-4a60-8e12-c6338131cd03\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " Oct 30 18:14:50.904468 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.900326 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-workload-certs\") pod \"b162d38e-8491-4a60-8e12-c6338131cd03\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " Oct 30 18:14:50.916868 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.905312 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:14:50.942865 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.905351 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:14:50.943325 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.916386 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-workload-certs" pod "b162d38e-8491-4a60-8e12-c6338131cd03" (UID: "b162d38e-8491-4a60-8e12-c6338131cd03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:14:50.943465 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.919983 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-istio-envoy" pod "b162d38e-8491-4a60-8e12-c6338131cd03" (UID: "b162d38e-8491-4a60-8e12-c6338131cd03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:14:50.947573 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.925648 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-credential-socket" pod "b162d38e-8491-4a60-8e12-c6338131cd03" (UID: "b162d38e-8491-4a60-8e12-c6338131cd03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:14:50.947731 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.917724 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b162d38e-8491-4a60-8e12-c6338131cd03-istio-token\" (UniqueName: \"kubernetes.io/projected/b162d38e-8491-4a60-8e12-c6338131cd03-istio-token\") pod \"b162d38e-8491-4a60-8e12-c6338131cd03\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " Oct 30 18:14:50.953822 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.953793 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b162d38e-8491-4a60-8e12-c6338131cd03-kube-api-access-k9wd7\" (UniqueName: \"kubernetes.io/projected/b162d38e-8491-4a60-8e12-c6338131cd03-kube-api-access-k9wd7\") pod \"b162d38e-8491-4a60-8e12-c6338131cd03\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " Oct 30 18:14:50.963576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.954298 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b162d38e-8491-4a60-8e12-c6338131cd03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b162d38e-8491-4a60-8e12-c6338131cd03-istio-podinfo\") pod \"b162d38e-8491-4a60-8e12-c6338131cd03\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " Oct 30 18:14:50.963795 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.963774 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-workload-socket\") pod \"b162d38e-8491-4a60-8e12-c6338131cd03\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " Oct 30 18:14:50.974921 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.974899 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-istio-data\") pod \"b162d38e-8491-4a60-8e12-c6338131cd03\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " Oct 30 18:14:50.978673 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.978651 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b162d38e-8491-4a60-8e12-c6338131cd03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b162d38e-8491-4a60-8e12-c6338131cd03-istiod-ca-cert\") pod \"b162d38e-8491-4a60-8e12-c6338131cd03\" (UID: \"b162d38e-8491-4a60-8e12-c6338131cd03\") " Oct 30 18:14:50.979673 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.979650 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:14:50.981733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.981717 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:14:50.981862 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.981850 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:14:50.981946 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.959953 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b162d38e-8491-4a60-8e12-c6338131cd03-istio-podinfo" pod "b162d38e-8491-4a60-8e12-c6338131cd03" (UID: "b162d38e-8491-4a60-8e12-c6338131cd03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:14:50.982048 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.960075 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b162d38e-8491-4a60-8e12-c6338131cd03-kube-api-access-k9wd7" pod "b162d38e-8491-4a60-8e12-c6338131cd03" (UID: "b162d38e-8491-4a60-8e12-c6338131cd03"). InnerVolumeSpecName "kube-api-access-k9wd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:50.982145 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.967571 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-workload-socket" pod "b162d38e-8491-4a60-8e12-c6338131cd03" (UID: "b162d38e-8491-4a60-8e12-c6338131cd03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:14:50.982229 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.968652 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b162d38e-8491-4a60-8e12-c6338131cd03-istio-token" pod "b162d38e-8491-4a60-8e12-c6338131cd03" (UID: "b162d38e-8491-4a60-8e12-c6338131cd03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:50.982309 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.978575 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-istio-data" pod "b162d38e-8491-4a60-8e12-c6338131cd03" (UID: "b162d38e-8491-4a60-8e12-c6338131cd03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:14:50.982392 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.981671 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b162d38e-8491-4a60-8e12-c6338131cd03-istiod-ca-cert" pod "b162d38e-8491-4a60-8e12-c6338131cd03" (UID: "b162d38e-8491-4a60-8e12-c6338131cd03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:50.982543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:50.982490 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:14:51.047532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.042690 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" event={"ID":"b162d38e-8491-4a60-8e12-c6338131cd03","Type":"ContainerDied","Data":"f206ebfae50ef2582b2d6e3e7520cce1a7dd6372c7ef42c8d8999e8b409b9e63"} Oct 30 18:14:51.047532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.042762 4603 scope.go:122] "RemoveContainer" containerID="c8dd7267e455e5ee7f017b159860937aaf5fde66bdb3c288958441f62faeb01f" Oct 30 18:14:51.054532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.054326 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:14:51.055690 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.055621 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl" Oct 30 18:14:51.083919 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.083884 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b162d38e-8491-4a60-8e12-c6338131cd03-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:14:51.084101 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.084087 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9wd7\" (UniqueName: \"kubernetes.io/projected/b162d38e-8491-4a60-8e12-c6338131cd03-kube-api-access-k9wd7\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:14:51.084239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.084223 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b162d38e-8491-4a60-8e12-c6338131cd03-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:14:51.084369 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.084355 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:14:51.084534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.084520 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b162d38e-8491-4a60-8e12-c6338131cd03-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:14:51.084799 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.084785 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b162d38e-8491-4a60-8e12-c6338131cd03-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:14:51.320527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.288493 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:51.333911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.333858 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl"] Oct 30 18:14:51.351349 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.351321 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-4gvxl"] Oct 30 18:14:51.373083 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.368819 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:51.379492 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.378784 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:14:51.869414 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.868661 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:51.874079 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.873995 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:51.877373 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.877334 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b162d38e-8491-4a60-8e12-c6338131cd03" path="/var/lib/kubelet/pods/b162d38e-8491-4a60-8e12-c6338131cd03/volumes" Oct 30 18:14:51.878350 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.878328 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:14:51.885930 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.885912 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:14:51.887086 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.886788 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:14:51.887617 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.887582 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:14:51.887942 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.887922 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:14:51.956897 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.956865 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2"] Oct 30 18:14:51.958176 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:14:51.958150 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb952b3_3096_442b_8ab6_2909cee2c021.slice/crio-16e533922b4625bb49b92b4523ba2bc9e0c02b2da4a1110e19a0cec366eea5eb WatchSource:0}: Error finding container 16e533922b4625bb49b92b4523ba2bc9e0c02b2da4a1110e19a0cec366eea5eb: Status 404 returned error can't find the container with id 16e533922b4625bb49b92b4523ba2bc9e0c02b2da4a1110e19a0cec366eea5eb Oct 30 18:14:51.958554 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.958536 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:14:51.960440 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.960405 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:51.960589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.960487 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:51.960589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:51.960551 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:52.046529 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:14:52.037796 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:14:52.058747 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:52.058699 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" event={"ID":"3bb952b3-3096-442b-8ab6-2909cee2c021","Type":"ContainerStarted","Data":"16e533922b4625bb49b92b4523ba2bc9e0c02b2da4a1110e19a0cec366eea5eb"} Oct 30 18:14:52.951387 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:52.951348 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:53.095545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:53.095437 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" event={"ID":"3bb952b3-3096-442b-8ab6-2909cee2c021","Type":"ContainerStarted","Data":"29f2db7a9fed86224bd7bbf5685d606ede4d5b3a0c5a6fff0f036271f11bff75"} Oct 30 18:14:53.184266 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:53.184211 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" podStartSLOduration=5.184195046 podStartE2EDuration="5.184195046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:53.181435382 +0000 UTC m=+1353.897005256" watchObservedRunningTime="2026-10-30 18:14:53.184195046 +0000 UTC m=+1353.899764897" Oct 30 18:14:53.719338 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:53.719309 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:53.744628 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:53.744564 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:54.119497 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:54.119468 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:54.120389 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:54.120371 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:14:54.623012 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:54.622977 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm"] Oct 30 18:14:54.749547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:54.747690 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:54.924898 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:54.924700 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:55.238112 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.238027 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw"] Oct 30 18:14:55.239003 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.238984 4603 cpu_manager.go:414] "containerMap: removing container" podUID="b162d38e-8491-4a60-8e12-c6338131cd03" containerName="istio-proxy" Oct 30 18:14:55.239070 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.239006 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b162d38e-8491-4a60-8e12-c6338131cd03" containerName="istio-proxy" Oct 30 18:14:55.239070 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.239030 4603 cpu_manager.go:414] "containerMap: removing container" podUID="b162d38e-8491-4a60-8e12-c6338131cd03" containerName="istio-proxy" Oct 30 18:14:55.239070 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.239038 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b162d38e-8491-4a60-8e12-c6338131cd03" containerName="istio-proxy" Oct 30 18:14:55.239205 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.239192 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="b162d38e-8491-4a60-8e12-c6338131cd03" containerName="istio-proxy" Oct 30 18:14:55.239246 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.239209 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="b162d38e-8491-4a60-8e12-c6338131cd03" containerName="istio-proxy" Oct 30 18:14:55.242560 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.242538 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.272630 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.272598 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw"] Oct 30 18:14:55.367558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.356441 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-token\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.367558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.356576 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-credential-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.367558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.356660 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-workload-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.367558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.356794 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-envoy\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.367558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.356847 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-workload-certs\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.367558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.356932 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istiod-ca-cert\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.367558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.356977 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-podinfo\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.367558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.357053 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j25w\" (UniqueName: \"kubernetes.io/projected/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-kube-api-access-4j25w\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.367558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.357097 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-data\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.471693 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.471657 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-workload-certs\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.471947 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.471928 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istiod-ca-cert\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.472071 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.472056 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-podinfo\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.472198 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.472186 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j25w\" (UniqueName: \"kubernetes.io/projected/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-kube-api-access-4j25w\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.472293 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.472280 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-data\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.472423 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.472412 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-token\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.472600 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.472582 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-credential-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.472795 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.472778 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-workload-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.473030 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.473013 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-envoy\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.479087 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.477039 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-workload-certs\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.479087 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.477834 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-data\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.479087 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.478478 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istiod-ca-cert\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.479087 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.478815 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-credential-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.479524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.479380 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-workload-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.554556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.544349 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-envoy\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.554556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.547545 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-token\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.583061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.567286 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-podinfo\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.613085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.613046 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j25w\" (UniqueName: \"kubernetes.io/projected/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-kube-api-access-4j25w\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:55.863106 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:55.863011 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:56.202485 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:14:56.202434 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59611c8e_86d7_44a9_b52f_a1a8ce2f0720.slice/crio-610c2f8d8eb17c61d111dc9c74e006f1cf00393c2ab5076ec5e3b81683cbd750 WatchSource:0}: Error finding container 610c2f8d8eb17c61d111dc9c74e006f1cf00393c2ab5076ec5e3b81683cbd750: Status 404 returned error can't find the container with id 610c2f8d8eb17c61d111dc9c74e006f1cf00393c2ab5076ec5e3b81683cbd750 Oct 30 18:14:56.208488 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:56.208451 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:56.208671 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:56.208578 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:56.208671 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:56.208621 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:56.215813 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:56.213393 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw"] Oct 30 18:14:56.269051 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:56.269012 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:56.829856 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:56.829826 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:56.992216 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:56.992167 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:57.147892 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:57.147801 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" event={"ID":"59611c8e-86d7-44a9-b52f-a1a8ce2f0720","Type":"ContainerStarted","Data":"015a207e3260a204db163e81230625cdaaccc33fd343f6e14e5eb24fbb959c7f"} Oct 30 18:14:57.147892 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:57.147866 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" event={"ID":"59611c8e-86d7-44a9-b52f-a1a8ce2f0720","Type":"ContainerStarted","Data":"610c2f8d8eb17c61d111dc9c74e006f1cf00393c2ab5076ec5e3b81683cbd750"} Oct 30 18:14:57.203552 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:57.203480 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" podStartSLOduration=2.203465036 podStartE2EDuration="2.203465036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:57.202719602 +0000 UTC m=+1357.918289475" watchObservedRunningTime="2026-10-30 18:14:57.203465036 +0000 UTC m=+1357.919034909" Oct 30 18:14:57.214618 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:57.214584 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:57.586527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:57.586468 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:57.864129 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:57.864049 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:14:57.866118 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:57.866091 4603 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 18:14:57.866245 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:57.866146 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" podUID="59611c8e-86d7-44a9-b52f-a1a8ce2f0720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 18:14:58.487652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:58.487615 4603 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:58.487836 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:58.487668 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" podUID="b4eac5a8-12c7-41b6-a8ac-8e570386586c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:58.864363 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:58.864330 4603 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 18:14:58.864919 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:58.864385 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" podUID="59611c8e-86d7-44a9-b52f-a1a8ce2f0720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 18:14:59.221289 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:59.221217 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:59.624523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:59.624463 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm" podUID="f158a045-fa10-4b24-ae77-829b4a70b423" containerName="payload-processing" containerID="cri-o://1ccef2a829da1f10efb8bd90ef4cbf7ca4e4c32c011d2e750bd0b5de3d2f09a5" gracePeriod=25 Oct 30 18:14:59.866010 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:59.864482 4603 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 18:14:59.866010 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:59.864573 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" podUID="59611c8e-86d7-44a9-b52f-a1a8ce2f0720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 18:14:59.967360 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:14:59.967337 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm" Oct 30 18:15:00.053985 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.053957 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f158a045-fa10-4b24-ae77-829b4a70b423-kube-api-access-jj42w\" (UniqueName: \"kubernetes.io/projected/f158a045-fa10-4b24-ae77-829b4a70b423-kube-api-access-jj42w\") pod \"f158a045-fa10-4b24-ae77-829b4a70b423\" (UID: \"f158a045-fa10-4b24-ae77-829b4a70b423\") " Oct 30 18:15:00.054186 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.054104 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f158a045-fa10-4b24-ae77-829b4a70b423-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f158a045-fa10-4b24-ae77-829b4a70b423-plugins-config-volume\") pod \"f158a045-fa10-4b24-ae77-829b4a70b423\" (UID: \"f158a045-fa10-4b24-ae77-829b4a70b423\") " Oct 30 18:15:00.054722 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.054692 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f158a045-fa10-4b24-ae77-829b4a70b423-plugins-config-volume" pod "f158a045-fa10-4b24-ae77-829b4a70b423" (UID: "f158a045-fa10-4b24-ae77-829b4a70b423"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:00.056381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.056359 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f158a045-fa10-4b24-ae77-829b4a70b423-kube-api-access-jj42w" pod "f158a045-fa10-4b24-ae77-829b4a70b423" (UID: "f158a045-fa10-4b24-ae77-829b4a70b423"). InnerVolumeSpecName "kube-api-access-jj42w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:00.150309 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.150222 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl"] Oct 30 18:15:00.151222 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.151203 4603 cpu_manager.go:414] "containerMap: removing container" podUID="f158a045-fa10-4b24-ae77-829b4a70b423" containerName="payload-processing" Oct 30 18:15:00.151322 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.151225 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f158a045-fa10-4b24-ae77-829b4a70b423" containerName="payload-processing" Oct 30 18:15:00.151392 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.151371 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="f158a045-fa10-4b24-ae77-829b4a70b423" containerName="payload-processing" Oct 30 18:15:00.154342 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.154322 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl" Oct 30 18:15:00.155561 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.155535 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj42w\" (UniqueName: \"kubernetes.io/projected/f158a045-fa10-4b24-ae77-829b4a70b423-kube-api-access-jj42w\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:00.155657 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.155567 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f158a045-fa10-4b24-ae77-829b4a70b423-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:00.167974 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.167951 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn"] Oct 30 18:15:00.175217 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.175190 4603 generic.go:356] "Generic (PLEG): container finished" podID="f158a045-fa10-4b24-ae77-829b4a70b423" containerID="1ccef2a829da1f10efb8bd90ef4cbf7ca4e4c32c011d2e750bd0b5de3d2f09a5" exitCode=0 Oct 30 18:15:00.175414 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.175390 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm" Oct 30 18:15:00.175850 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.175829 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm" event={"ID":"f158a045-fa10-4b24-ae77-829b4a70b423","Type":"ContainerDied","Data":"1ccef2a829da1f10efb8bd90ef4cbf7ca4e4c32c011d2e750bd0b5de3d2f09a5"} Oct 30 18:15:00.176001 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.175983 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm" event={"ID":"f158a045-fa10-4b24-ae77-829b4a70b423","Type":"ContainerDied","Data":"84e83ead594f41411ea063882b90b8fe5860de19a336981d3f819b11d3bd7af2"} Oct 30 18:15:00.176120 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.176107 4603 scope.go:122] "RemoveContainer" containerID="1ccef2a829da1f10efb8bd90ef4cbf7ca4e4c32c011d2e750bd0b5de3d2f09a5" Oct 30 18:15:00.176396 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.176382 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn" Oct 30 18:15:00.179693 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.179657 4603 generic.go:356] "Generic (PLEG): container finished" podID="b4eac5a8-12c7-41b6-a8ac-8e570386586c" containerID="33046b182173b53bab6743bc3bfae8d29b429cb5333ea2642320c68b70a438cc" exitCode=0 Oct 30 18:15:00.180015 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.179996 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" event={"ID":"b4eac5a8-12c7-41b6-a8ac-8e570386586c","Type":"ContainerDied","Data":"33046b182173b53bab6743bc3bfae8d29b429cb5333ea2642320c68b70a438cc"} Oct 30 18:15:00.180834 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.180809 4603 scope.go:122] "RemoveContainer" containerID="33046b182173b53bab6743bc3bfae8d29b429cb5333ea2642320c68b70a438cc" Oct 30 18:15:00.183625 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.183556 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:00.183727 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.183638 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:00.183727 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.183688 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:00.183727 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.183708 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk"] Oct 30 18:15:00.188166 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.187837 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk" Oct 30 18:15:00.210401 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.189114 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl"] Oct 30 18:15:00.224819 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.218517 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk"] Oct 30 18:15:00.235265 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.234459 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn"] Oct 30 18:15:00.268994 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.268972 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:15:00.269772 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.269748 4603 scope.go:122] "RemoveContainer" containerID="1ccef2a829da1f10efb8bd90ef4cbf7ca4e4c32c011d2e750bd0b5de3d2f09a5" Oct 30 18:15:00.270271 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:15:00.270244 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ccef2a829da1f10efb8bd90ef4cbf7ca4e4c32c011d2e750bd0b5de3d2f09a5\": container with ID starting with 1ccef2a829da1f10efb8bd90ef4cbf7ca4e4c32c011d2e750bd0b5de3d2f09a5 not found: ID does not exist" containerID="1ccef2a829da1f10efb8bd90ef4cbf7ca4e4c32c011d2e750bd0b5de3d2f09a5" Oct 30 18:15:00.270432 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.270403 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ccef2a829da1f10efb8bd90ef4cbf7ca4e4c32c011d2e750bd0b5de3d2f09a5"} err="failed to get container status \"1ccef2a829da1f10efb8bd90ef4cbf7ca4e4c32c011d2e750bd0b5de3d2f09a5\": rpc error: code = NotFound desc = could not find container \"1ccef2a829da1f10efb8bd90ef4cbf7ca4e4c32c011d2e750bd0b5de3d2f09a5\": container with ID starting with 1ccef2a829da1f10efb8bd90ef4cbf7ca4e4c32c011d2e750bd0b5de3d2f09a5 not found: ID does not exist" Oct 30 18:15:00.362298 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.362268 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61cec429-6d83-430a-9c51-2942c12dac9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk\" (UID: \"61cec429-6d83-430a-9c51-2942c12dac9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk" Oct 30 18:15:00.362402 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.362306 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4c5t\" (UniqueName: \"kubernetes.io/projected/5fdac8a8-5038-402c-a3f2-c4393f772caa-kube-api-access-m4c5t\") pod \"maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn\" (UID: \"5fdac8a8-5038-402c-a3f2-c4393f772caa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn" Oct 30 18:15:00.362568 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.362551 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a773b93a-1df6-40c3-8bac-d2a8e7a4421a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl\" (UID: \"a773b93a-1df6-40c3-8bac-d2a8e7a4421a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl" Oct 30 18:15:00.362624 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.362584 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj2gp\" (UniqueName: \"kubernetes.io/projected/a773b93a-1df6-40c3-8bac-d2a8e7a4421a-kube-api-access-vj2gp\") pod \"maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl\" (UID: \"a773b93a-1df6-40c3-8bac-d2a8e7a4421a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl" Oct 30 18:15:00.362925 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.362772 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fdac8a8-5038-402c-a3f2-c4393f772caa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn\" (UID: \"5fdac8a8-5038-402c-a3f2-c4393f772caa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn" Oct 30 18:15:00.362925 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.362849 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfn9n\" (UniqueName: \"kubernetes.io/projected/61cec429-6d83-430a-9c51-2942c12dac9a-kube-api-access-pfn9n\") pod \"maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk\" (UID: \"61cec429-6d83-430a-9c51-2942c12dac9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk" Oct 30 18:15:00.426097 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.425984 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm"] Oct 30 18:15:00.431624 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.430640 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0498d2-556c6fbc4-qdbxm"] Oct 30 18:15:00.467363 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.464809 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a773b93a-1df6-40c3-8bac-d2a8e7a4421a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl\" (UID: \"a773b93a-1df6-40c3-8bac-d2a8e7a4421a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl" Oct 30 18:15:00.467363 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.464857 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj2gp\" (UniqueName: \"kubernetes.io/projected/a773b93a-1df6-40c3-8bac-d2a8e7a4421a-kube-api-access-vj2gp\") pod \"maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl\" (UID: \"a773b93a-1df6-40c3-8bac-d2a8e7a4421a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl" Oct 30 18:15:00.467363 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.464981 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fdac8a8-5038-402c-a3f2-c4393f772caa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn\" (UID: \"5fdac8a8-5038-402c-a3f2-c4393f772caa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn" Oct 30 18:15:00.467363 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.465050 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfn9n\" (UniqueName: \"kubernetes.io/projected/61cec429-6d83-430a-9c51-2942c12dac9a-kube-api-access-pfn9n\") pod \"maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk\" (UID: \"61cec429-6d83-430a-9c51-2942c12dac9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk" Oct 30 18:15:00.467363 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.465149 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61cec429-6d83-430a-9c51-2942c12dac9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk\" (UID: \"61cec429-6d83-430a-9c51-2942c12dac9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk" Oct 30 18:15:00.467363 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.465175 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4c5t\" (UniqueName: \"kubernetes.io/projected/5fdac8a8-5038-402c-a3f2-c4393f772caa-kube-api-access-m4c5t\") pod \"maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn\" (UID: \"5fdac8a8-5038-402c-a3f2-c4393f772caa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn" Oct 30 18:15:00.467363 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.466324 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a773b93a-1df6-40c3-8bac-d2a8e7a4421a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl\" (UID: \"a773b93a-1df6-40c3-8bac-d2a8e7a4421a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl" Oct 30 18:15:00.467363 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.466449 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fdac8a8-5038-402c-a3f2-c4393f772caa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn\" (UID: \"5fdac8a8-5038-402c-a3f2-c4393f772caa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn" Oct 30 18:15:00.467363 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.467320 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61cec429-6d83-430a-9c51-2942c12dac9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk\" (UID: \"61cec429-6d83-430a-9c51-2942c12dac9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk" Oct 30 18:15:00.543642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.542598 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4c5t\" (UniqueName: \"kubernetes.io/projected/5fdac8a8-5038-402c-a3f2-c4393f772caa-kube-api-access-m4c5t\") pod \"maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn\" (UID: \"5fdac8a8-5038-402c-a3f2-c4393f772caa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn" Oct 30 18:15:00.544077 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.544050 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfn9n\" (UniqueName: \"kubernetes.io/projected/61cec429-6d83-430a-9c51-2942c12dac9a-kube-api-access-pfn9n\") pod \"maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk\" (UID: \"61cec429-6d83-430a-9c51-2942c12dac9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk" Oct 30 18:15:00.544269 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.544237 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj2gp\" (UniqueName: \"kubernetes.io/projected/a773b93a-1df6-40c3-8bac-d2a8e7a4421a-kube-api-access-vj2gp\") pod \"maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl\" (UID: \"a773b93a-1df6-40c3-8bac-d2a8e7a4421a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl" Oct 30 18:15:00.553741 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.553714 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn" Oct 30 18:15:00.578403 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.578383 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk" Oct 30 18:15:00.774984 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.774958 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl" Oct 30 18:15:00.864223 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.864152 4603 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 18:15:00.864223 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:00.864195 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" podUID="59611c8e-86d7-44a9-b52f-a1a8ce2f0720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 18:15:01.020110 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:01.020080 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn"] Oct 30 18:15:01.025680 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:01.025650 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk"] Oct 30 18:15:01.029628 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:15:01.029600 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61cec429_6d83_430a_9c51_2942c12dac9a.slice/crio-b78805c5d4ee546c636c0032d953d83288a50225f8f4627cb66b0e8d9918c149 WatchSource:0}: Error finding container b78805c5d4ee546c636c0032d953d83288a50225f8f4627cb66b0e8d9918c149: Status 404 returned error can't find the container with id b78805c5d4ee546c636c0032d953d83288a50225f8f4627cb66b0e8d9918c149 Oct 30 18:15:01.169919 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:01.169893 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:01.241585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:01.241200 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl"] Oct 30 18:15:01.280534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:01.270359 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" event={"ID":"b4eac5a8-12c7-41b6-a8ac-8e570386586c","Type":"ContainerStarted","Data":"031e180f66ad5e401b3641040d346ab3515586a60ca1e676aceb66ff721e3769"} Oct 30 18:15:01.293833 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:01.291913 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk" event={"ID":"61cec429-6d83-430a-9c51-2942c12dac9a","Type":"ContainerStarted","Data":"b78805c5d4ee546c636c0032d953d83288a50225f8f4627cb66b0e8d9918c149"} Oct 30 18:15:01.313245 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:01.313215 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:15:01.313439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:01.313349 4603 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:15:01.313439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:01.313391 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" podUID="b4eac5a8-12c7-41b6-a8ac-8e570386586c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 18:15:01.313754 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:01.313724 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn" event={"ID":"5fdac8a8-5038-402c-a3f2-c4393f772caa","Type":"ContainerStarted","Data":"7de393fdf50889df278b06d73412a2fa8f695d539b7efe02cb51ef935deeb210"} Oct 30 18:15:01.354725 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:01.348967 4603 generic.go:356] "Generic (PLEG): container finished" podID="c7e02b19-7f85-4e69-8a62-f5436875c9e4" containerID="896109ffdae1339171426490195b8633d98b45ff0736bc0b5614fafbf0653bb3" exitCode=0 Oct 30 18:15:01.354725 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:01.349072 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" event={"ID":"c7e02b19-7f85-4e69-8a62-f5436875c9e4","Type":"ContainerDied","Data":"896109ffdae1339171426490195b8633d98b45ff0736bc0b5614fafbf0653bb3"} Oct 30 18:15:01.354725 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:01.349602 4603 scope.go:122] "RemoveContainer" containerID="896109ffdae1339171426490195b8633d98b45ff0736bc0b5614fafbf0653bb3" Oct 30 18:15:01.386521 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:01.386468 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:01.386803 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:01.386778 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:01.386960 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:01.386940 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:01.444700 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:01.444201 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f158a045-fa10-4b24-ae77-829b4a70b423" path="/var/lib/kubelet/pods/f158a045-fa10-4b24-ae77-829b4a70b423/volumes" Oct 30 18:15:01.864248 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:01.864220 4603 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 18:15:01.864357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:01.864268 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" podUID="59611c8e-86d7-44a9-b52f-a1a8ce2f0720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 18:15:02.314529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:02.312876 4603 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:15:02.314529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:02.312937 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" podUID="b4eac5a8-12c7-41b6-a8ac-8e570386586c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 18:15:02.445585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:02.444356 4603 generic.go:356] "Generic (PLEG): container finished" podID="61cec429-6d83-430a-9c51-2942c12dac9a" containerID="bc8535c792b5e11978955bf8f618373ec0ad7643e1dc6d89101c9618bdf163da" exitCode=60 Oct 30 18:15:02.445585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:02.444600 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk" event={"ID":"61cec429-6d83-430a-9c51-2942c12dac9a","Type":"ContainerDied","Data":"bc8535c792b5e11978955bf8f618373ec0ad7643e1dc6d89101c9618bdf163da"} Oct 30 18:15:02.445585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:02.445343 4603 scope.go:122] "RemoveContainer" containerID="bc8535c792b5e11978955bf8f618373ec0ad7643e1dc6d89101c9618bdf163da" Oct 30 18:15:02.449011 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:02.448991 4603 generic.go:356] "Generic (PLEG): container finished" podID="5fdac8a8-5038-402c-a3f2-c4393f772caa" containerID="ae53dd2de61cc6936a92d0f66863fd4dda31b1eb2a4697c3f29d7c62887a6347" exitCode=60 Oct 30 18:15:02.449414 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:02.449069 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn" event={"ID":"5fdac8a8-5038-402c-a3f2-c4393f772caa","Type":"ContainerDied","Data":"ae53dd2de61cc6936a92d0f66863fd4dda31b1eb2a4697c3f29d7c62887a6347"} Oct 30 18:15:02.449633 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:02.449613 4603 scope.go:122] "RemoveContainer" containerID="ae53dd2de61cc6936a92d0f66863fd4dda31b1eb2a4697c3f29d7c62887a6347" Oct 30 18:15:02.453636 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:02.452844 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl" event={"ID":"a773b93a-1df6-40c3-8bac-d2a8e7a4421a","Type":"ContainerStarted","Data":"486cf38ca3b8c8a620e418a0bce130a188fde9303a4c3ddd9135a6c774470686"} Oct 30 18:15:02.453636 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:02.452892 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl" event={"ID":"a773b93a-1df6-40c3-8bac-d2a8e7a4421a","Type":"ContainerStarted","Data":"bec2c660e82d6ed77b1f00c936a174695c03a690e614b5a369763e7943f75fa1"} Oct 30 18:15:02.458331 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:02.457134 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" event={"ID":"c7e02b19-7f85-4e69-8a62-f5436875c9e4","Type":"ContainerStarted","Data":"fe117c3f6d40e6a1339a3fd69e05599730fab8de44440109e43bd6c23b31e4cd"} Oct 30 18:15:02.462914 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:02.462850 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:15:02.463121 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:02.463058 4603 patch_prober.go:29] interesting pod/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:15:02.463121 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:02.463092 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" podUID="c7e02b19-7f85-4e69-8a62-f5436875c9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:15:02.893746 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:02.893465 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:02.917163 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:02.895690 4603 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 18:15:02.917163 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:02.895745 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" podUID="59611c8e-86d7-44a9-b52f-a1a8ce2f0720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 18:15:03.082350 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:03.081624 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl" podStartSLOduration=3.081601462 podStartE2EDuration="3.081601462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:02.775111066 +0000 UTC m=+1363.490680978" watchObservedRunningTime="2026-10-30 18:15:03.081601462 +0000 UTC m=+1363.797171348" Oct 30 18:15:03.320542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:03.317075 4603 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:15:03.320542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:03.317134 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" podUID="b4eac5a8-12c7-41b6-a8ac-8e570386586c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 18:15:03.463329 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:03.463236 4603 patch_prober.go:29] interesting pod/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:15:03.463329 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:03.463286 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" podUID="c7e02b19-7f85-4e69-8a62-f5436875c9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:15:03.466426 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:03.466136 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk" event={"ID":"61cec429-6d83-430a-9c51-2942c12dac9a","Type":"ContainerStarted","Data":"06a6cb1dd792e46af885bbce4f63dea56194054661cbc9a3c7f536d3d6ecabc9"} Oct 30 18:15:03.469277 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:03.469256 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn" event={"ID":"5fdac8a8-5038-402c-a3f2-c4393f772caa","Type":"ContainerStarted","Data":"26ac01254ac6d41dd550a09795a8e05a7456dead584a3f2fcb53adf0d707592d"} Oct 30 18:15:03.471335 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:03.471315 4603 generic.go:356] "Generic (PLEG): container finished" podID="a773b93a-1df6-40c3-8bac-d2a8e7a4421a" containerID="486cf38ca3b8c8a620e418a0bce130a188fde9303a4c3ddd9135a6c774470686" exitCode=60 Oct 30 18:15:03.471528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:03.471394 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl" event={"ID":"a773b93a-1df6-40c3-8bac-d2a8e7a4421a","Type":"ContainerDied","Data":"486cf38ca3b8c8a620e418a0bce130a188fde9303a4c3ddd9135a6c774470686"} Oct 30 18:15:03.471673 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:03.471614 4603 scope.go:122] "RemoveContainer" containerID="486cf38ca3b8c8a620e418a0bce130a188fde9303a4c3ddd9135a6c774470686" Oct 30 18:15:03.559784 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:03.552222 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk" podStartSLOduration=3.552201969 podStartE2EDuration="3.552201969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:03.506649383 +0000 UTC m=+1364.222219282" watchObservedRunningTime="2026-10-30 18:15:03.552201969 +0000 UTC m=+1364.267771844" Oct 30 18:15:03.565826 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:03.565749 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn" podStartSLOduration=3.565732639 podStartE2EDuration="3.565732639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:03.563373153 +0000 UTC m=+1364.278943033" watchObservedRunningTime="2026-10-30 18:15:03.565732639 +0000 UTC m=+1364.281302513" Oct 30 18:15:03.869455 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:03.869429 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:15:03.869889 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:03.869871 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:15:03.870730 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:03.870710 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:15:04.229579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:04.225253 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:04.318527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:04.309929 4603 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:15:04.318527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:04.309995 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" podUID="b4eac5a8-12c7-41b6-a8ac-8e570386586c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 18:15:04.375529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:04.371949 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:04.463162 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:04.463129 4603 patch_prober.go:29] interesting pod/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:15:04.463311 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:04.463178 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" podUID="c7e02b19-7f85-4e69-8a62-f5436875c9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:15:04.483301 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:04.483218 4603 generic.go:356] "Generic (PLEG): container finished" podID="61cec429-6d83-430a-9c51-2942c12dac9a" containerID="06a6cb1dd792e46af885bbce4f63dea56194054661cbc9a3c7f536d3d6ecabc9" exitCode=60 Oct 30 18:15:04.483301 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:04.483253 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk" event={"ID":"61cec429-6d83-430a-9c51-2942c12dac9a","Type":"ContainerDied","Data":"06a6cb1dd792e46af885bbce4f63dea56194054661cbc9a3c7f536d3d6ecabc9"} Oct 30 18:15:04.483483 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:04.483317 4603 scope.go:122] "RemoveContainer" containerID="bc8535c792b5e11978955bf8f618373ec0ad7643e1dc6d89101c9618bdf163da" Oct 30 18:15:04.483648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:04.483634 4603 scope.go:122] "RemoveContainer" containerID="06a6cb1dd792e46af885bbce4f63dea56194054661cbc9a3c7f536d3d6ecabc9" Oct 30 18:15:04.483934 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:15:04.483909 4603 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk_odh-ai-gateway-infra(61cec429-6d83-430a-9c51-2942c12dac9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk" podUID="61cec429-6d83-430a-9c51-2942c12dac9a" Oct 30 18:15:04.486795 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:04.486765 4603 generic.go:356] "Generic (PLEG): container finished" podID="5fdac8a8-5038-402c-a3f2-c4393f772caa" containerID="26ac01254ac6d41dd550a09795a8e05a7456dead584a3f2fcb53adf0d707592d" exitCode=60 Oct 30 18:15:04.486915 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:04.486840 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn" event={"ID":"5fdac8a8-5038-402c-a3f2-c4393f772caa","Type":"ContainerDied","Data":"26ac01254ac6d41dd550a09795a8e05a7456dead584a3f2fcb53adf0d707592d"} Oct 30 18:15:04.487156 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:04.487141 4603 scope.go:122] "RemoveContainer" containerID="26ac01254ac6d41dd550a09795a8e05a7456dead584a3f2fcb53adf0d707592d" Oct 30 18:15:04.487409 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:15:04.487390 4603 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn_odh-ai-gateway-infra(5fdac8a8-5038-402c-a3f2-c4393f772caa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn" podUID="5fdac8a8-5038-402c-a3f2-c4393f772caa" Oct 30 18:15:04.564710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:04.564678 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9"] Oct 30 18:15:04.565163 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:04.565136 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" podUID="8697996a-d257-4a4e-9c3c-e372ee1b0726" containerName="istio-proxy" containerID="cri-o://69ebf57a80b52c2f71efea3baa0180f3d7ffea8756fe47c4802e0999d3e17183" gracePeriod=30 Oct 30 18:15:04.573316 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:04.573234 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl" event={"ID":"a773b93a-1df6-40c3-8bac-d2a8e7a4421a","Type":"ContainerStarted","Data":"a476514ee41122122f80a9c62f33c954311fd2efef7d8e3f020eb3e37e56d971"} Oct 30 18:15:04.584551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:04.584358 4603 scope.go:122] "RemoveContainer" containerID="ae53dd2de61cc6936a92d0f66863fd4dda31b1eb2a4697c3f29d7c62887a6347" Oct 30 18:15:04.587058 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:04.586345 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:04.778780 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:04.776785 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:05.328529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:05.325482 4603 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:15:05.328529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:05.325565 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" podUID="b4eac5a8-12c7-41b6-a8ac-8e570386586c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 18:15:05.463396 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:05.463369 4603 patch_prober.go:29] interesting pod/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:15:05.463892 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:05.463867 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" podUID="c7e02b19-7f85-4e69-8a62-f5436875c9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:15:05.583470 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:05.583385 4603 scope.go:122] "RemoveContainer" containerID="06a6cb1dd792e46af885bbce4f63dea56194054661cbc9a3c7f536d3d6ecabc9" Oct 30 18:15:05.583834 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:15:05.583800 4603 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk_odh-ai-gateway-infra(61cec429-6d83-430a-9c51-2942c12dac9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk" podUID="61cec429-6d83-430a-9c51-2942c12dac9a" Oct 30 18:15:05.610541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:05.586608 4603 scope.go:122] "RemoveContainer" containerID="26ac01254ac6d41dd550a09795a8e05a7456dead584a3f2fcb53adf0d707592d" Oct 30 18:15:05.610541 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:15:05.586847 4603 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn_odh-ai-gateway-infra(5fdac8a8-5038-402c-a3f2-c4393f772caa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn" podUID="5fdac8a8-5038-402c-a3f2-c4393f772caa" Oct 30 18:15:05.610541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:05.610118 4603 generic.go:356] "Generic (PLEG): container finished" podID="a773b93a-1df6-40c3-8bac-d2a8e7a4421a" containerID="a476514ee41122122f80a9c62f33c954311fd2efef7d8e3f020eb3e37e56d971" exitCode=60 Oct 30 18:15:05.618540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:05.614790 4603 scope.go:122] "RemoveContainer" containerID="a476514ee41122122f80a9c62f33c954311fd2efef7d8e3f020eb3e37e56d971" Oct 30 18:15:05.618540 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:15:05.615163 4603 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl_odh-ai-gateway-infra(a773b93a-1df6-40c3-8bac-d2a8e7a4421a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl" podUID="a773b93a-1df6-40c3-8bac-d2a8e7a4421a" Oct 30 18:15:05.618540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:05.615214 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl" event={"ID":"a773b93a-1df6-40c3-8bac-d2a8e7a4421a","Type":"ContainerDied","Data":"a476514ee41122122f80a9c62f33c954311fd2efef7d8e3f020eb3e37e56d971"} Oct 30 18:15:05.618540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:05.615264 4603 scope.go:122] "RemoveContainer" containerID="486cf38ca3b8c8a620e418a0bce130a188fde9303a4c3ddd9135a6c774470686" Oct 30 18:15:05.727569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:05.710651 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:05.912585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:05.899156 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:06.311536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:06.309073 4603 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:15:06.311536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:06.309138 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" podUID="b4eac5a8-12c7-41b6-a8ac-8e570386586c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 18:15:06.462429 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:06.462402 4603 patch_prober.go:29] interesting pod/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:15:06.462652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:06.462623 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" podUID="c7e02b19-7f85-4e69-8a62-f5436875c9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:15:06.646709 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:06.646624 4603 scope.go:122] "RemoveContainer" containerID="a476514ee41122122f80a9c62f33c954311fd2efef7d8e3f020eb3e37e56d971" Oct 30 18:15:06.647676 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:15:06.647652 4603 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl_odh-ai-gateway-infra(a773b93a-1df6-40c3-8bac-d2a8e7a4421a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl" podUID="a773b93a-1df6-40c3-8bac-d2a8e7a4421a" Oct 30 18:15:06.664711 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:06.664683 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:07.311150 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:07.311115 4603 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:15:07.311759 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:07.311730 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" podUID="b4eac5a8-12c7-41b6-a8ac-8e570386586c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 18:15:07.438644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:07.438497 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:07.462280 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:07.462252 4603 patch_prober.go:29] interesting pod/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:15:07.462465 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:07.462307 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" podUID="c7e02b19-7f85-4e69-8a62-f5436875c9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:15:07.553185 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:07.553151 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p"] Oct 30 18:15:07.557618 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:07.557593 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p" Oct 30 18:15:07.574694 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:07.574670 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c8f010-dockercfg-h2wlb\"" Oct 30 18:15:07.631541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:07.606598 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c8f010\"" Oct 30 18:15:07.631541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:07.608306 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnjd5\" (UniqueName: \"kubernetes.io/projected/9d600055-7b4d-4586-9ca5-1305adb1983c-kube-api-access-cnjd5\") pod \"payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p\" (UID: \"9d600055-7b4d-4586-9ca5-1305adb1983c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p" Oct 30 18:15:07.631541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:07.608361 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d600055-7b4d-4586-9ca5-1305adb1983c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p\" (UID: \"9d600055-7b4d-4586-9ca5-1305adb1983c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p" Oct 30 18:15:07.631541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:07.610101 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:07.675604 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:07.663655 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p"] Oct 30 18:15:07.731691 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:07.731653 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnjd5\" (UniqueName: \"kubernetes.io/projected/9d600055-7b4d-4586-9ca5-1305adb1983c-kube-api-access-cnjd5\") pod \"payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p\" (UID: \"9d600055-7b4d-4586-9ca5-1305adb1983c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p" Oct 30 18:15:07.757795 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:07.757624 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d600055-7b4d-4586-9ca5-1305adb1983c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p\" (UID: \"9d600055-7b4d-4586-9ca5-1305adb1983c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p" Oct 30 18:15:07.786759 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:07.759691 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d600055-7b4d-4586-9ca5-1305adb1983c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p\" (UID: \"9d600055-7b4d-4586-9ca5-1305adb1983c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p" Oct 30 18:15:08.074727 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:08.074693 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnjd5\" (UniqueName: \"kubernetes.io/projected/9d600055-7b4d-4586-9ca5-1305adb1983c-kube-api-access-cnjd5\") pod \"payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p\" (UID: \"9d600055-7b4d-4586-9ca5-1305adb1983c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p" Oct 30 18:15:08.171094 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:08.171062 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p" Oct 30 18:15:08.322783 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:08.322753 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:15:08.325598 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:08.323919 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:15:08.325598 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:08.325119 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:15:08.417468 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:08.415034 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:15:08.462437 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:08.462388 4603 patch_prober.go:29] interesting pod/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:15:08.462628 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:08.462450 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" podUID="c7e02b19-7f85-4e69-8a62-f5436875c9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:15:08.517918 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:08.517881 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:08.519788 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:08.519764 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p"] Oct 30 18:15:08.520820 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:15:08.520771 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d600055_7b4d_4586_9ca5_1305adb1983c.slice/crio-77dfe2b1a553c617d772714af2f62f01844b5891004473cafa84e0116a92149e WatchSource:0}: Error finding container 77dfe2b1a553c617d772714af2f62f01844b5891004473cafa84e0116a92149e: Status 404 returned error can't find the container with id 77dfe2b1a553c617d772714af2f62f01844b5891004473cafa84e0116a92149e Oct 30 18:15:09.075565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:09.075525 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p" event={"ID":"9d600055-7b4d-4586-9ca5-1305adb1983c","Type":"ContainerStarted","Data":"9f2d0808c67aa843b9cf6e0b3906cd69b578d712a93a33e0b6f04e834ca956f7"} Oct 30 18:15:09.075998 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:09.075588 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p" event={"ID":"9d600055-7b4d-4586-9ca5-1305adb1983c","Type":"ContainerStarted","Data":"77dfe2b1a553c617d772714af2f62f01844b5891004473cafa84e0116a92149e"} Oct 30 18:15:09.075998 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:09.075630 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p" Oct 30 18:15:09.117852 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:09.117804 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p" podStartSLOduration=2.117791587 podStartE2EDuration="2.117791587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:09.114569963 +0000 UTC m=+1369.830139835" watchObservedRunningTime="2026-10-30 18:15:09.117791587 +0000 UTC m=+1369.833361517" Oct 30 18:15:09.466880 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:09.466806 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:15:09.467930 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:09.467907 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:15:09.945218 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:09.945197 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:15:10.048706 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.048680 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-credential-socket\") pod \"8697996a-d257-4a4e-9c3c-e372ee1b0726\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " Oct 30 18:15:10.048864 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.048724 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-token\" (UniqueName: \"kubernetes.io/projected/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-token\") pod \"8697996a-d257-4a4e-9c3c-e372ee1b0726\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " Oct 30 18:15:10.048864 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.048790 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-data\") pod \"8697996a-d257-4a4e-9c3c-e372ee1b0726\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " Oct 30 18:15:10.048864 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.048837 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-workload-certs\") pod \"8697996a-d257-4a4e-9c3c-e372ee1b0726\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " Oct 30 18:15:10.048997 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.048919 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-envoy\") pod \"8697996a-d257-4a4e-9c3c-e372ee1b0726\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " Oct 30 18:15:10.048997 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.048948 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8697996a-d257-4a4e-9c3c-e372ee1b0726-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8697996a-d257-4a4e-9c3c-e372ee1b0726-istiod-ca-cert\") pod \"8697996a-d257-4a4e-9c3c-e372ee1b0726\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " Oct 30 18:15:10.048997 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.048952 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-credential-socket" pod "8697996a-d257-4a4e-9c3c-e372ee1b0726" (UID: "8697996a-d257-4a4e-9c3c-e372ee1b0726"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:15:10.048997 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.048990 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-workload-socket\") pod \"8697996a-d257-4a4e-9c3c-e372ee1b0726\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " Oct 30 18:15:10.049244 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.049070 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-podinfo\") pod \"8697996a-d257-4a4e-9c3c-e372ee1b0726\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " Oct 30 18:15:10.049244 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.049123 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-data" pod "8697996a-d257-4a4e-9c3c-e372ee1b0726" (UID: "8697996a-d257-4a4e-9c3c-e372ee1b0726"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:15:10.049244 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.049181 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-workload-certs" pod "8697996a-d257-4a4e-9c3c-e372ee1b0726" (UID: "8697996a-d257-4a4e-9c3c-e372ee1b0726"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:15:10.049244 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.049215 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8697996a-d257-4a4e-9c3c-e372ee1b0726-kube-api-access-5fglx\" (UniqueName: \"kubernetes.io/projected/8697996a-d257-4a4e-9c3c-e372ee1b0726-kube-api-access-5fglx\") pod \"8697996a-d257-4a4e-9c3c-e372ee1b0726\" (UID: \"8697996a-d257-4a4e-9c3c-e372ee1b0726\") " Oct 30 18:15:10.049454 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.049330 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-workload-socket" pod "8697996a-d257-4a4e-9c3c-e372ee1b0726" (UID: "8697996a-d257-4a4e-9c3c-e372ee1b0726"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:15:10.049454 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.049359 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8697996a-d257-4a4e-9c3c-e372ee1b0726-istiod-ca-cert" pod "8697996a-d257-4a4e-9c3c-e372ee1b0726" (UID: "8697996a-d257-4a4e-9c3c-e372ee1b0726"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:10.050148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.050125 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:10.050263 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.050155 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:10.050263 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.050171 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:10.050263 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.050185 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8697996a-d257-4a4e-9c3c-e372ee1b0726-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:10.050263 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.050199 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:10.051434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.051408 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-podinfo" pod "8697996a-d257-4a4e-9c3c-e372ee1b0726" (UID: "8697996a-d257-4a4e-9c3c-e372ee1b0726"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:15:10.051434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.051415 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-token" pod "8697996a-d257-4a4e-9c3c-e372ee1b0726" (UID: "8697996a-d257-4a4e-9c3c-e372ee1b0726"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:10.051434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.051419 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-envoy" pod "8697996a-d257-4a4e-9c3c-e372ee1b0726" (UID: "8697996a-d257-4a4e-9c3c-e372ee1b0726"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:15:10.051862 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.051838 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8697996a-d257-4a4e-9c3c-e372ee1b0726-kube-api-access-5fglx" pod "8697996a-d257-4a4e-9c3c-e372ee1b0726" (UID: "8697996a-d257-4a4e-9c3c-e372ee1b0726"). InnerVolumeSpecName "kube-api-access-5fglx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:10.084947 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.084921 4603 generic.go:356] "Generic (PLEG): container finished" podID="8697996a-d257-4a4e-9c3c-e372ee1b0726" containerID="69ebf57a80b52c2f71efea3baa0180f3d7ffea8756fe47c4802e0999d3e17183" exitCode=0 Oct 30 18:15:10.085311 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.084963 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" event={"ID":"8697996a-d257-4a4e-9c3c-e372ee1b0726","Type":"ContainerDied","Data":"69ebf57a80b52c2f71efea3baa0180f3d7ffea8756fe47c4802e0999d3e17183"} Oct 30 18:15:10.085311 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.084977 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" Oct 30 18:15:10.085311 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.085020 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9" event={"ID":"8697996a-d257-4a4e-9c3c-e372ee1b0726","Type":"ContainerDied","Data":"d558ffa5c3a0949c694fef114919dc5acf05d5fe3a23db7a8352c6fb8157d312"} Oct 30 18:15:10.085311 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.085036 4603 scope.go:122] "RemoveContainer" containerID="69ebf57a80b52c2f71efea3baa0180f3d7ffea8756fe47c4802e0999d3e17183" Oct 30 18:15:10.096470 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.096453 4603 scope.go:122] "RemoveContainer" containerID="6c609ca59027de3e803458ad36e66f0b143761b51b24b1c73c20c5a00e55b4a1" Oct 30 18:15:10.105543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.105491 4603 scope.go:122] "RemoveContainer" containerID="69ebf57a80b52c2f71efea3baa0180f3d7ffea8756fe47c4802e0999d3e17183" Oct 30 18:15:10.105774 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:15:10.105759 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69ebf57a80b52c2f71efea3baa0180f3d7ffea8756fe47c4802e0999d3e17183\": container with ID starting with 69ebf57a80b52c2f71efea3baa0180f3d7ffea8756fe47c4802e0999d3e17183 not found: ID does not exist" containerID="69ebf57a80b52c2f71efea3baa0180f3d7ffea8756fe47c4802e0999d3e17183" Oct 30 18:15:10.105821 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.105783 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69ebf57a80b52c2f71efea3baa0180f3d7ffea8756fe47c4802e0999d3e17183"} err="failed to get container status \"69ebf57a80b52c2f71efea3baa0180f3d7ffea8756fe47c4802e0999d3e17183\": rpc error: code = NotFound desc = could not find container \"69ebf57a80b52c2f71efea3baa0180f3d7ffea8756fe47c4802e0999d3e17183\": container with ID starting with 69ebf57a80b52c2f71efea3baa0180f3d7ffea8756fe47c4802e0999d3e17183 not found: ID does not exist" Oct 30 18:15:10.105821 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.105799 4603 scope.go:122] "RemoveContainer" containerID="6c609ca59027de3e803458ad36e66f0b143761b51b24b1c73c20c5a00e55b4a1" Oct 30 18:15:10.106076 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:15:10.106058 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c609ca59027de3e803458ad36e66f0b143761b51b24b1c73c20c5a00e55b4a1\": container with ID starting with 6c609ca59027de3e803458ad36e66f0b143761b51b24b1c73c20c5a00e55b4a1 not found: ID does not exist" containerID="6c609ca59027de3e803458ad36e66f0b143761b51b24b1c73c20c5a00e55b4a1" Oct 30 18:15:10.106131 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.106084 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c609ca59027de3e803458ad36e66f0b143761b51b24b1c73c20c5a00e55b4a1"} err="failed to get container status \"6c609ca59027de3e803458ad36e66f0b143761b51b24b1c73c20c5a00e55b4a1\": rpc error: code = NotFound desc = could not find container \"6c609ca59027de3e803458ad36e66f0b143761b51b24b1c73c20c5a00e55b4a1\": container with ID starting with 6c609ca59027de3e803458ad36e66f0b143761b51b24b1c73c20c5a00e55b4a1 not found: ID does not exist" Oct 30 18:15:10.151826 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.151805 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:10.151826 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.151829 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:10.151942 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.151837 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8697996a-d257-4a4e-9c3c-e372ee1b0726-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:10.151942 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.151845 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fglx\" (UniqueName: \"kubernetes.io/projected/8697996a-d257-4a4e-9c3c-e372ee1b0726-kube-api-access-5fglx\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:10.171302 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.171280 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9"] Oct 30 18:15:10.219115 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.219091 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4c8e1a-openshift-default-574b7f7d9c-n8xh9"] Oct 30 18:15:10.349456 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:10.349382 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:11.416328 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:11.416285 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8697996a-d257-4a4e-9c3c-e372ee1b0726" path="/var/lib/kubelet/pods/8697996a-d257-4a4e-9c3c-e372ee1b0726/volumes" Oct 30 18:15:12.083810 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:12.083772 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:13.867284 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:13.867247 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:14.430874 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:14.430847 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:15.134614 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:15.134582 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl"] Oct 30 18:15:15.135190 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:15.135176 4603 cpu_manager.go:414] "containerMap: removing container" podUID="8697996a-d257-4a4e-9c3c-e372ee1b0726" containerName="istio-proxy" Oct 30 18:15:15.135236 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:15.135190 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8697996a-d257-4a4e-9c3c-e372ee1b0726" containerName="istio-proxy" Oct 30 18:15:15.135306 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:15.135298 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="8697996a-d257-4a4e-9c3c-e372ee1b0726" containerName="istio-proxy" Oct 30 18:15:15.138640 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:15.138620 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl" Oct 30 18:15:15.153261 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:15.153241 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:15:15.170968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:15.170943 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl"] Oct 30 18:15:15.186717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:15.186687 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e48ed8-4a07-45cb-ac86-959f20ae1c35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl\" (UID: \"28e48ed8-4a07-45cb-ac86-959f20ae1c35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl" Oct 30 18:15:15.186997 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:15.186976 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e48ed8-4a07-45cb-ac86-959f20ae1c35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl\" (UID: \"28e48ed8-4a07-45cb-ac86-959f20ae1c35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl" Oct 30 18:15:15.187121 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:15.187102 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dsq4\" (UniqueName: \"kubernetes.io/projected/28e48ed8-4a07-45cb-ac86-959f20ae1c35-kube-api-access-6dsq4\") pod \"maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl\" (UID: \"28e48ed8-4a07-45cb-ac86-959f20ae1c35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl" Oct 30 18:15:15.288162 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:15.288137 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e48ed8-4a07-45cb-ac86-959f20ae1c35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl\" (UID: \"28e48ed8-4a07-45cb-ac86-959f20ae1c35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl" Oct 30 18:15:15.288301 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:15.288194 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dsq4\" (UniqueName: \"kubernetes.io/projected/28e48ed8-4a07-45cb-ac86-959f20ae1c35-kube-api-access-6dsq4\") pod \"maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl\" (UID: \"28e48ed8-4a07-45cb-ac86-959f20ae1c35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl" Oct 30 18:15:15.288301 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:15.288282 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e48ed8-4a07-45cb-ac86-959f20ae1c35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl\" (UID: \"28e48ed8-4a07-45cb-ac86-959f20ae1c35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl" Oct 30 18:15:15.288939 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:15.288914 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e48ed8-4a07-45cb-ac86-959f20ae1c35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl\" (UID: \"28e48ed8-4a07-45cb-ac86-959f20ae1c35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl" Oct 30 18:15:15.290728 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:15.290704 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e48ed8-4a07-45cb-ac86-959f20ae1c35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl\" (UID: \"28e48ed8-4a07-45cb-ac86-959f20ae1c35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl" Oct 30 18:15:15.311536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:15.311487 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dsq4\" (UniqueName: \"kubernetes.io/projected/28e48ed8-4a07-45cb-ac86-959f20ae1c35-kube-api-access-6dsq4\") pod \"maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl\" (UID: \"28e48ed8-4a07-45cb-ac86-959f20ae1c35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl" Oct 30 18:15:15.437211 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:15.437135 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn"] Oct 30 18:15:15.450936 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:15.450907 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl" Oct 30 18:15:15.469643 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:15.469615 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:15.714012 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:15.713621 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn" Oct 30 18:15:16.065525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:16.028538 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fdac8a8-5038-402c-a3f2-c4393f772caa-kube-api-access-m4c5t\" (UniqueName: \"kubernetes.io/projected/5fdac8a8-5038-402c-a3f2-c4393f772caa-kube-api-access-m4c5t\") pod \"5fdac8a8-5038-402c-a3f2-c4393f772caa\" (UID: \"5fdac8a8-5038-402c-a3f2-c4393f772caa\") " Oct 30 18:15:16.114009 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:16.113974 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fdac8a8-5038-402c-a3f2-c4393f772caa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fdac8a8-5038-402c-a3f2-c4393f772caa-maas-api-service-ca\") pod \"5fdac8a8-5038-402c-a3f2-c4393f772caa\" (UID: \"5fdac8a8-5038-402c-a3f2-c4393f772caa\") " Oct 30 18:15:16.115109 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:16.086641 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fdac8a8-5038-402c-a3f2-c4393f772caa-kube-api-access-m4c5t" pod "5fdac8a8-5038-402c-a3f2-c4393f772caa" (UID: "5fdac8a8-5038-402c-a3f2-c4393f772caa"). InnerVolumeSpecName "kube-api-access-m4c5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:16.115550 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:16.115527 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fdac8a8-5038-402c-a3f2-c4393f772caa-maas-api-service-ca" pod "5fdac8a8-5038-402c-a3f2-c4393f772caa" (UID: "5fdac8a8-5038-402c-a3f2-c4393f772caa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:16.173551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:16.155455 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl"] Oct 30 18:15:16.191447 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:16.191413 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:16.199084 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:16.199045 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl" event={"ID":"28e48ed8-4a07-45cb-ac86-959f20ae1c35","Type":"ContainerStarted","Data":"be3be8025d243cf166a6e9c50da9e65cc4331294caf80cf7bcdb74037652ba73"} Oct 30 18:15:16.201296 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:16.201266 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn" event={"ID":"5fdac8a8-5038-402c-a3f2-c4393f772caa","Type":"ContainerDied","Data":"7de393fdf50889df278b06d73412a2fa8f695d539b7efe02cb51ef935deeb210"} Oct 30 18:15:16.201488 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:16.201470 4603 scope.go:122] "RemoveContainer" containerID="26ac01254ac6d41dd550a09795a8e05a7456dead584a3f2fcb53adf0d707592d" Oct 30 18:15:16.201795 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:16.201775 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn" Oct 30 18:15:16.223896 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:16.221816 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4c5t\" (UniqueName: \"kubernetes.io/projected/5fdac8a8-5038-402c-a3f2-c4393f772caa-kube-api-access-m4c5t\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:16.223896 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:16.221851 4603 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fdac8a8-5038-402c-a3f2-c4393f772caa-maas-api-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:16.278362 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:16.278321 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn"] Oct 30 18:15:16.287610 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:16.287579 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0498d2-29889735-pwlgn"] Oct 30 18:15:16.341183 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:16.341104 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:16.410223 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:16.408270 4603 scope.go:122] "RemoveContainer" containerID="06a6cb1dd792e46af885bbce4f63dea56194054661cbc9a3c7f536d3d6ecabc9" Oct 30 18:15:16.495712 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:16.495682 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:16.809800 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:16.809768 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:17.209515 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:17.209422 4603 generic.go:356] "Generic (PLEG): container finished" podID="28e48ed8-4a07-45cb-ac86-959f20ae1c35" containerID="d8540aad3df7dc33d7a250392ce1302f71e36d23a2dec67321d6cb14725c4e98" exitCode=0 Oct 30 18:15:17.209944 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:17.209522 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl" event={"ID":"28e48ed8-4a07-45cb-ac86-959f20ae1c35","Type":"ContainerDied","Data":"d8540aad3df7dc33d7a250392ce1302f71e36d23a2dec67321d6cb14725c4e98"} Oct 30 18:15:17.211353 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:17.211330 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk" event={"ID":"61cec429-6d83-430a-9c51-2942c12dac9a","Type":"ContainerStarted","Data":"0bfb2586af42809d3bdd2fa38c4d070558c67c31196e17bb5d6603261da20fdb"} Oct 30 18:15:17.410322 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:17.410288 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fdac8a8-5038-402c-a3f2-c4393f772caa" path="/var/lib/kubelet/pods/5fdac8a8-5038-402c-a3f2-c4393f772caa/volumes" Oct 30 18:15:18.156982 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:18.156933 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:18.300360 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:18.299090 4603 generic.go:356] "Generic (PLEG): container finished" podID="61cec429-6d83-430a-9c51-2942c12dac9a" containerID="0bfb2586af42809d3bdd2fa38c4d070558c67c31196e17bb5d6603261da20fdb" exitCode=60 Oct 30 18:15:18.300360 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:18.299167 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk" event={"ID":"61cec429-6d83-430a-9c51-2942c12dac9a","Type":"ContainerDied","Data":"0bfb2586af42809d3bdd2fa38c4d070558c67c31196e17bb5d6603261da20fdb"} Oct 30 18:15:18.300360 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:18.299245 4603 scope.go:122] "RemoveContainer" containerID="06a6cb1dd792e46af885bbce4f63dea56194054661cbc9a3c7f536d3d6ecabc9" Oct 30 18:15:18.300360 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:18.299877 4603 scope.go:122] "RemoveContainer" containerID="0bfb2586af42809d3bdd2fa38c4d070558c67c31196e17bb5d6603261da20fdb" Oct 30 18:15:18.300360 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:15:18.300255 4603 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk_odh-ai-gateway-infra(61cec429-6d83-430a-9c51-2942c12dac9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk" podUID="61cec429-6d83-430a-9c51-2942c12dac9a" Oct 30 18:15:18.301862 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:18.301835 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk"] Oct 30 18:15:18.427570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:18.404640 4603 scope.go:122] "RemoveContainer" containerID="a476514ee41122122f80a9c62f33c954311fd2efef7d8e3f020eb3e37e56d971" Oct 30 18:15:18.490660 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:18.490639 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl" Oct 30 18:15:18.582666 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:18.582635 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28e48ed8-4a07-45cb-ac86-959f20ae1c35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e48ed8-4a07-45cb-ac86-959f20ae1c35-maas-api-xks-ca\") pod \"28e48ed8-4a07-45cb-ac86-959f20ae1c35\" (UID: \"28e48ed8-4a07-45cb-ac86-959f20ae1c35\") " Oct 30 18:15:18.582992 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:18.582709 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28e48ed8-4a07-45cb-ac86-959f20ae1c35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e48ed8-4a07-45cb-ac86-959f20ae1c35-maas-api-openshift-ca\") pod \"28e48ed8-4a07-45cb-ac86-959f20ae1c35\" (UID: \"28e48ed8-4a07-45cb-ac86-959f20ae1c35\") " Oct 30 18:15:18.583091 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:18.583035 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e48ed8-4a07-45cb-ac86-959f20ae1c35-kube-api-access-6dsq4\" (UniqueName: \"kubernetes.io/projected/28e48ed8-4a07-45cb-ac86-959f20ae1c35-kube-api-access-6dsq4\") pod \"28e48ed8-4a07-45cb-ac86-959f20ae1c35\" (UID: \"28e48ed8-4a07-45cb-ac86-959f20ae1c35\") " Oct 30 18:15:18.584139 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:18.584111 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28e48ed8-4a07-45cb-ac86-959f20ae1c35-maas-api-openshift-ca" pod "28e48ed8-4a07-45cb-ac86-959f20ae1c35" (UID: "28e48ed8-4a07-45cb-ac86-959f20ae1c35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:18.594658 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:18.594629 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28e48ed8-4a07-45cb-ac86-959f20ae1c35-maas-api-xks-ca" pod "28e48ed8-4a07-45cb-ac86-959f20ae1c35" (UID: "28e48ed8-4a07-45cb-ac86-959f20ae1c35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:18.594782 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:18.594764 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e48ed8-4a07-45cb-ac86-959f20ae1c35-kube-api-access-6dsq4" pod "28e48ed8-4a07-45cb-ac86-959f20ae1c35" (UID: "28e48ed8-4a07-45cb-ac86-959f20ae1c35"). InnerVolumeSpecName "kube-api-access-6dsq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:18.685077 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:18.685012 4603 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e48ed8-4a07-45cb-ac86-959f20ae1c35-maas-api-xks-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:18.685077 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:18.685039 4603 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e48ed8-4a07-45cb-ac86-959f20ae1c35-maas-api-openshift-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:18.685077 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:18.685049 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dsq4\" (UniqueName: \"kubernetes.io/projected/28e48ed8-4a07-45cb-ac86-959f20ae1c35-kube-api-access-6dsq4\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:19.307548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:19.307518 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl" Oct 30 18:15:19.307548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:19.307534 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl" event={"ID":"28e48ed8-4a07-45cb-ac86-959f20ae1c35","Type":"ContainerDied","Data":"be3be8025d243cf166a6e9c50da9e65cc4331294caf80cf7bcdb74037652ba73"} Oct 30 18:15:19.308004 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:19.307622 4603 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be3be8025d243cf166a6e9c50da9e65cc4331294caf80cf7bcdb74037652ba73" Oct 30 18:15:19.311632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:19.311605 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl" event={"ID":"a773b93a-1df6-40c3-8bac-d2a8e7a4421a","Type":"ContainerStarted","Data":"daa2fa0f85e14b92a292649921a6372eeb42c3af7196200dbd6f8415bc10d678"} Oct 30 18:15:19.473784 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:19.473759 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk" Oct 30 18:15:19.598097 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:19.598020 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61cec429-6d83-430a-9c51-2942c12dac9a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61cec429-6d83-430a-9c51-2942c12dac9a-maas-api-service-ca\") pod \"61cec429-6d83-430a-9c51-2942c12dac9a\" (UID: \"61cec429-6d83-430a-9c51-2942c12dac9a\") " Oct 30 18:15:19.598222 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:19.598129 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61cec429-6d83-430a-9c51-2942c12dac9a-kube-api-access-pfn9n\" (UniqueName: \"kubernetes.io/projected/61cec429-6d83-430a-9c51-2942c12dac9a-kube-api-access-pfn9n\") pod \"61cec429-6d83-430a-9c51-2942c12dac9a\" (UID: \"61cec429-6d83-430a-9c51-2942c12dac9a\") " Oct 30 18:15:19.598440 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:19.598416 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61cec429-6d83-430a-9c51-2942c12dac9a-maas-api-service-ca" pod "61cec429-6d83-430a-9c51-2942c12dac9a" (UID: "61cec429-6d83-430a-9c51-2942c12dac9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:19.598744 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:19.598721 4603 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61cec429-6d83-430a-9c51-2942c12dac9a-maas-api-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:19.600475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:19.600447 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61cec429-6d83-430a-9c51-2942c12dac9a-kube-api-access-pfn9n" pod "61cec429-6d83-430a-9c51-2942c12dac9a" (UID: "61cec429-6d83-430a-9c51-2942c12dac9a"). InnerVolumeSpecName "kube-api-access-pfn9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:19.700359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:19.700330 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfn9n\" (UniqueName: \"kubernetes.io/projected/61cec429-6d83-430a-9c51-2942c12dac9a-kube-api-access-pfn9n\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:19.989590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:19.989486 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:20.088882 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:20.088856 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p" Oct 30 18:15:20.171119 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:20.169259 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:20.323628 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:20.323596 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:20.340161 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:20.340110 4603 generic.go:356] "Generic (PLEG): container finished" podID="a773b93a-1df6-40c3-8bac-d2a8e7a4421a" containerID="daa2fa0f85e14b92a292649921a6372eeb42c3af7196200dbd6f8415bc10d678" exitCode=60 Oct 30 18:15:20.343118 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:20.343081 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl" event={"ID":"a773b93a-1df6-40c3-8bac-d2a8e7a4421a","Type":"ContainerDied","Data":"daa2fa0f85e14b92a292649921a6372eeb42c3af7196200dbd6f8415bc10d678"} Oct 30 18:15:20.344420 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:20.343407 4603 scope.go:122] "RemoveContainer" containerID="a476514ee41122122f80a9c62f33c954311fd2efef7d8e3f020eb3e37e56d971" Oct 30 18:15:20.344739 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:20.344711 4603 scope.go:122] "RemoveContainer" containerID="daa2fa0f85e14b92a292649921a6372eeb42c3af7196200dbd6f8415bc10d678" Oct 30 18:15:20.345391 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:15:20.345363 4603 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl_odh-ai-gateway-infra(a773b93a-1df6-40c3-8bac-d2a8e7a4421a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl" podUID="a773b93a-1df6-40c3-8bac-d2a8e7a4421a" Oct 30 18:15:20.347278 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:20.347248 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk" event={"ID":"61cec429-6d83-430a-9c51-2942c12dac9a","Type":"ContainerDied","Data":"b78805c5d4ee546c636c0032d953d83288a50225f8f4627cb66b0e8d9918c149"} Oct 30 18:15:20.347414 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:20.347395 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk" Oct 30 18:15:20.362297 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:20.360814 4603 scope.go:122] "RemoveContainer" containerID="0bfb2586af42809d3bdd2fa38c4d070558c67c31196e17bb5d6603261da20fdb" Oct 30 18:15:20.417754 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:20.417728 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl"] Oct 30 18:15:20.421555 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:20.421523 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk"] Oct 30 18:15:20.427225 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:20.427205 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0bdf03-29889735-mx6bk"] Oct 30 18:15:21.442634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:21.442595 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61cec429-6d83-430a-9c51-2942c12dac9a" path="/var/lib/kubelet/pods/61cec429-6d83-430a-9c51-2942c12dac9a/volumes" Oct 30 18:15:21.562992 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:21.562969 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl" Oct 30 18:15:21.674303 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:21.674228 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a773b93a-1df6-40c3-8bac-d2a8e7a4421a-kube-api-access-vj2gp\" (UniqueName: \"kubernetes.io/projected/a773b93a-1df6-40c3-8bac-d2a8e7a4421a-kube-api-access-vj2gp\") pod \"a773b93a-1df6-40c3-8bac-d2a8e7a4421a\" (UID: \"a773b93a-1df6-40c3-8bac-d2a8e7a4421a\") " Oct 30 18:15:21.674458 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:21.674431 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a773b93a-1df6-40c3-8bac-d2a8e7a4421a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a773b93a-1df6-40c3-8bac-d2a8e7a4421a-maas-api-service-ca\") pod \"a773b93a-1df6-40c3-8bac-d2a8e7a4421a\" (UID: \"a773b93a-1df6-40c3-8bac-d2a8e7a4421a\") " Oct 30 18:15:21.674881 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:21.674854 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a773b93a-1df6-40c3-8bac-d2a8e7a4421a-maas-api-service-ca" pod "a773b93a-1df6-40c3-8bac-d2a8e7a4421a" (UID: "a773b93a-1df6-40c3-8bac-d2a8e7a4421a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:21.675117 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:21.675102 4603 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a773b93a-1df6-40c3-8bac-d2a8e7a4421a-maas-api-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:21.676441 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:21.676417 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a773b93a-1df6-40c3-8bac-d2a8e7a4421a-kube-api-access-vj2gp" pod "a773b93a-1df6-40c3-8bac-d2a8e7a4421a" (UID: "a773b93a-1df6-40c3-8bac-d2a8e7a4421a"). InnerVolumeSpecName "kube-api-access-vj2gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:21.776792 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:21.776767 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj2gp\" (UniqueName: \"kubernetes.io/projected/a773b93a-1df6-40c3-8bac-d2a8e7a4421a-kube-api-access-vj2gp\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:21.861524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:21.859446 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:21.965383 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:15:21.965282 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:15:22.365331 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:22.365301 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl" Oct 30 18:15:22.365331 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:22.365321 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl" event={"ID":"a773b93a-1df6-40c3-8bac-d2a8e7a4421a","Type":"ContainerDied","Data":"bec2c660e82d6ed77b1f00c936a174695c03a690e614b5a369763e7943f75fa1"} Oct 30 18:15:22.365575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:22.365387 4603 scope.go:122] "RemoveContainer" containerID="daa2fa0f85e14b92a292649921a6372eeb42c3af7196200dbd6f8415bc10d678" Oct 30 18:15:22.418466 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:22.418441 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl"] Oct 30 18:15:22.430373 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:22.430347 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889735-wbkdl"] Oct 30 18:15:22.790268 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:22.790228 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:22.931876 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:22.931844 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:23.410455 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:23.410426 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a773b93a-1df6-40c3-8bac-d2a8e7a4421a" path="/var/lib/kubelet/pods/a773b93a-1df6-40c3-8bac-d2a8e7a4421a/volumes" Oct 30 18:15:23.434059 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:23.434026 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:25.041903 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:25.041874 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:25.714895 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:25.714860 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf"] Oct 30 18:15:25.860258 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:25.860225 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:26.007711 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:26.007621 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:26.838088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:26.838049 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:28.646528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:28.638332 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:30.263534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:30.255483 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:30.715688 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:30.715616 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf" podUID="44c65786-428f-4ac0-97db-020b7a909823" containerName="payload-pre-processing" containerID="cri-o://ca53c7a8674652eebfcd0e311f991ac7dae71ae81840e298a562d686c96b3d6f" gracePeriod=25 Oct 30 18:15:30.988053 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:30.988030 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf" Oct 30 18:15:31.060320 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:31.060291 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c65786-428f-4ac0-97db-020b7a909823-kube-api-access-t5xzs\" (UniqueName: \"kubernetes.io/projected/44c65786-428f-4ac0-97db-020b7a909823-kube-api-access-t5xzs\") pod \"44c65786-428f-4ac0-97db-020b7a909823\" (UID: \"44c65786-428f-4ac0-97db-020b7a909823\") " Oct 30 18:15:31.060464 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:31.060396 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44c65786-428f-4ac0-97db-020b7a909823-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44c65786-428f-4ac0-97db-020b7a909823-plugins-config-volume\") pod \"44c65786-428f-4ac0-97db-020b7a909823\" (UID: \"44c65786-428f-4ac0-97db-020b7a909823\") " Oct 30 18:15:31.060783 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:31.060761 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44c65786-428f-4ac0-97db-020b7a909823-plugins-config-volume" pod "44c65786-428f-4ac0-97db-020b7a909823" (UID: "44c65786-428f-4ac0-97db-020b7a909823"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:31.060984 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:31.060969 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44c65786-428f-4ac0-97db-020b7a909823-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:31.062745 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:31.062722 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c65786-428f-4ac0-97db-020b7a909823-kube-api-access-t5xzs" pod "44c65786-428f-4ac0-97db-020b7a909823" (UID: "44c65786-428f-4ac0-97db-020b7a909823"). InnerVolumeSpecName "kube-api-access-t5xzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:31.162475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:31.162445 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5xzs\" (UniqueName: \"kubernetes.io/projected/44c65786-428f-4ac0-97db-020b7a909823-kube-api-access-t5xzs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:31.454270 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:31.454243 4603 generic.go:356] "Generic (PLEG): container finished" podID="44c65786-428f-4ac0-97db-020b7a909823" containerID="ca53c7a8674652eebfcd0e311f991ac7dae71ae81840e298a562d686c96b3d6f" exitCode=0 Oct 30 18:15:31.454629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:31.454287 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf" Oct 30 18:15:31.454629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:31.454308 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf" event={"ID":"44c65786-428f-4ac0-97db-020b7a909823","Type":"ContainerDied","Data":"ca53c7a8674652eebfcd0e311f991ac7dae71ae81840e298a562d686c96b3d6f"} Oct 30 18:15:31.454629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:31.454343 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf" event={"ID":"44c65786-428f-4ac0-97db-020b7a909823","Type":"ContainerDied","Data":"c093e8cb062e18999c0dd9f5a8b649cbba5270326a1348f4ec0c0ede8e520021"} Oct 30 18:15:31.454629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:31.454362 4603 scope.go:122] "RemoveContainer" containerID="ca53c7a8674652eebfcd0e311f991ac7dae71ae81840e298a562d686c96b3d6f" Oct 30 18:15:31.464111 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:31.464083 4603 scope.go:122] "RemoveContainer" containerID="ca53c7a8674652eebfcd0e311f991ac7dae71ae81840e298a562d686c96b3d6f" Oct 30 18:15:31.464375 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:15:31.464352 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca53c7a8674652eebfcd0e311f991ac7dae71ae81840e298a562d686c96b3d6f\": container with ID starting with ca53c7a8674652eebfcd0e311f991ac7dae71ae81840e298a562d686c96b3d6f not found: ID does not exist" containerID="ca53c7a8674652eebfcd0e311f991ac7dae71ae81840e298a562d686c96b3d6f" Oct 30 18:15:31.464445 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:31.464387 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca53c7a8674652eebfcd0e311f991ac7dae71ae81840e298a562d686c96b3d6f"} err="failed to get container status \"ca53c7a8674652eebfcd0e311f991ac7dae71ae81840e298a562d686c96b3d6f\": rpc error: code = NotFound desc = could not find container \"ca53c7a8674652eebfcd0e311f991ac7dae71ae81840e298a562d686c96b3d6f\": container with ID starting with ca53c7a8674652eebfcd0e311f991ac7dae71ae81840e298a562d686c96b3d6f not found: ID does not exist" Oct 30 18:15:31.507733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:31.507703 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf"] Oct 30 18:15:31.517536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:31.517515 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0bdf03-55cf47bbf5-9grgf"] Oct 30 18:15:31.613041 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:31.613011 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:31.754575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:31.754469 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:31.902777 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:31.902723 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:33.071588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:33.071470 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd"] Oct 30 18:15:33.072170 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:33.071907 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" podUID="c7e02b19-7f85-4e69-8a62-f5436875c9e4" containerName="istio-proxy" containerID="cri-o://fe117c3f6d40e6a1339a3fd69e05599730fab8de44440109e43bd6c23b31e4cd" gracePeriod=30 Oct 30 18:15:33.129193 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:33.129156 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:33.451821 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:33.451277 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44c65786-428f-4ac0-97db-020b7a909823" path="/var/lib/kubelet/pods/44c65786-428f-4ac0-97db-020b7a909823/volumes" Oct 30 18:15:33.600982 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:33.600947 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:35.264319 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:35.261149 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:35.999768 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:35.999726 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:37.093525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:37.088812 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:38.336994 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.336971 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:15:38.441721 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.441640 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-envoy\") pod \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " Oct 30 18:15:38.441721 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.441675 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-workload-certs\") pod \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " Oct 30 18:15:38.441721 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.441722 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7e02b19-7f85-4e69-8a62-f5436875c9e4-kube-api-access-qrq9p\" (UniqueName: \"kubernetes.io/projected/c7e02b19-7f85-4e69-8a62-f5436875c9e4-kube-api-access-qrq9p\") pod \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " Oct 30 18:15:38.442026 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.441778 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-workload-socket\") pod \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " Oct 30 18:15:38.442026 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.441825 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-credential-socket\") pod \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " Oct 30 18:15:38.442026 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.441906 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-data\") pod \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " Oct 30 18:15:38.442026 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.441938 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istiod-ca-cert\") pod \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " Oct 30 18:15:38.442235 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.442041 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-workload-certs" pod "c7e02b19-7f85-4e69-8a62-f5436875c9e4" (UID: "c7e02b19-7f85-4e69-8a62-f5436875c9e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:15:38.442235 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.442067 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-token\" (UniqueName: \"kubernetes.io/projected/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-token\") pod \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " Oct 30 18:15:38.442235 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.442144 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-workload-socket" pod "c7e02b19-7f85-4e69-8a62-f5436875c9e4" (UID: "c7e02b19-7f85-4e69-8a62-f5436875c9e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:15:38.442235 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.442181 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-credential-socket" pod "c7e02b19-7f85-4e69-8a62-f5436875c9e4" (UID: "c7e02b19-7f85-4e69-8a62-f5436875c9e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:15:38.442433 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.442237 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-podinfo\") pod \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\" (UID: \"c7e02b19-7f85-4e69-8a62-f5436875c9e4\") " Oct 30 18:15:38.442433 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.442248 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-data" pod "c7e02b19-7f85-4e69-8a62-f5436875c9e4" (UID: "c7e02b19-7f85-4e69-8a62-f5436875c9e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:15:38.442561 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.442448 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istiod-ca-cert" pod "c7e02b19-7f85-4e69-8a62-f5436875c9e4" (UID: "c7e02b19-7f85-4e69-8a62-f5436875c9e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:38.442933 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.442904 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:38.443069 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.442936 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:38.443069 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.442951 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:38.443069 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.442966 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:38.443069 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.442979 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:38.444372 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.444351 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e02b19-7f85-4e69-8a62-f5436875c9e4-kube-api-access-qrq9p" pod "c7e02b19-7f85-4e69-8a62-f5436875c9e4" (UID: "c7e02b19-7f85-4e69-8a62-f5436875c9e4"). InnerVolumeSpecName "kube-api-access-qrq9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:38.444675 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.444657 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-envoy" pod "c7e02b19-7f85-4e69-8a62-f5436875c9e4" (UID: "c7e02b19-7f85-4e69-8a62-f5436875c9e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:15:38.444879 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.444865 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-token" pod "c7e02b19-7f85-4e69-8a62-f5436875c9e4" (UID: "c7e02b19-7f85-4e69-8a62-f5436875c9e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:38.445076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.445008 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-podinfo" pod "c7e02b19-7f85-4e69-8a62-f5436875c9e4" (UID: "c7e02b19-7f85-4e69-8a62-f5436875c9e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:15:38.516307 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.516283 4603 generic.go:356] "Generic (PLEG): container finished" podID="c7e02b19-7f85-4e69-8a62-f5436875c9e4" containerID="fe117c3f6d40e6a1339a3fd69e05599730fab8de44440109e43bd6c23b31e4cd" exitCode=0 Oct 30 18:15:38.516420 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.516331 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" Oct 30 18:15:38.516420 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.516369 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" event={"ID":"c7e02b19-7f85-4e69-8a62-f5436875c9e4","Type":"ContainerDied","Data":"fe117c3f6d40e6a1339a3fd69e05599730fab8de44440109e43bd6c23b31e4cd"} Oct 30 18:15:38.516497 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.516426 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd" event={"ID":"c7e02b19-7f85-4e69-8a62-f5436875c9e4","Type":"ContainerDied","Data":"f2bb89e11d0ebe1ed14efd976412546bf78ec72e1e1c4e05b1d854370eda8576"} Oct 30 18:15:38.516497 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.516451 4603 scope.go:122] "RemoveContainer" containerID="fe117c3f6d40e6a1339a3fd69e05599730fab8de44440109e43bd6c23b31e4cd" Oct 30 18:15:38.526607 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.526590 4603 scope.go:122] "RemoveContainer" containerID="896109ffdae1339171426490195b8633d98b45ff0736bc0b5614fafbf0653bb3" Oct 30 18:15:38.535367 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.535350 4603 scope.go:122] "RemoveContainer" containerID="fe117c3f6d40e6a1339a3fd69e05599730fab8de44440109e43bd6c23b31e4cd" Oct 30 18:15:38.535625 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:15:38.535609 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe117c3f6d40e6a1339a3fd69e05599730fab8de44440109e43bd6c23b31e4cd\": container with ID starting with fe117c3f6d40e6a1339a3fd69e05599730fab8de44440109e43bd6c23b31e4cd not found: ID does not exist" containerID="fe117c3f6d40e6a1339a3fd69e05599730fab8de44440109e43bd6c23b31e4cd" Oct 30 18:15:38.535682 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.535634 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe117c3f6d40e6a1339a3fd69e05599730fab8de44440109e43bd6c23b31e4cd"} err="failed to get container status \"fe117c3f6d40e6a1339a3fd69e05599730fab8de44440109e43bd6c23b31e4cd\": rpc error: code = NotFound desc = could not find container \"fe117c3f6d40e6a1339a3fd69e05599730fab8de44440109e43bd6c23b31e4cd\": container with ID starting with fe117c3f6d40e6a1339a3fd69e05599730fab8de44440109e43bd6c23b31e4cd not found: ID does not exist" Oct 30 18:15:38.535682 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.535650 4603 scope.go:122] "RemoveContainer" containerID="896109ffdae1339171426490195b8633d98b45ff0736bc0b5614fafbf0653bb3" Oct 30 18:15:38.535880 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:15:38.535863 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"896109ffdae1339171426490195b8633d98b45ff0736bc0b5614fafbf0653bb3\": container with ID starting with 896109ffdae1339171426490195b8633d98b45ff0736bc0b5614fafbf0653bb3 not found: ID does not exist" containerID="896109ffdae1339171426490195b8633d98b45ff0736bc0b5614fafbf0653bb3" Oct 30 18:15:38.535918 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.535888 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"896109ffdae1339171426490195b8633d98b45ff0736bc0b5614fafbf0653bb3"} err="failed to get container status \"896109ffdae1339171426490195b8633d98b45ff0736bc0b5614fafbf0653bb3\": rpc error: code = NotFound desc = could not find container \"896109ffdae1339171426490195b8633d98b45ff0736bc0b5614fafbf0653bb3\": container with ID starting with 896109ffdae1339171426490195b8633d98b45ff0736bc0b5614fafbf0653bb3 not found: ID does not exist" Oct 30 18:15:38.544015 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.543995 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:38.544101 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.544017 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:38.544101 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.544027 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrq9p\" (UniqueName: \"kubernetes.io/projected/c7e02b19-7f85-4e69-8a62-f5436875c9e4-kube-api-access-qrq9p\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:38.544101 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.544036 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7e02b19-7f85-4e69-8a62-f5436875c9e4-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:15:38.573318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.573296 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd"] Oct 30 18:15:38.578341 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.578320 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0498d2-openshift-default-7cc774fc4f-jdtrd"] Oct 30 18:15:38.651095 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:38.651072 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:39.414077 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:39.414035 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7e02b19-7f85-4e69-8a62-f5436875c9e4" path="/var/lib/kubelet/pods/c7e02b19-7f85-4e69-8a62-f5436875c9e4/volumes" Oct 30 18:15:40.362536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:40.359155 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:42.049417 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:42.049386 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:43.739945 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:43.739905 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:45.130933 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:45.130895 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:45.333192 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:45.333156 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:46.704742 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:46.704712 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:46.993622 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:46.989109 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:47.182077 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182047 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr"] Oct 30 18:15:47.182704 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182688 4603 cpu_manager.go:414] "containerMap: removing container" podUID="a773b93a-1df6-40c3-8bac-d2a8e7a4421a" containerName="cleanup" Oct 30 18:15:47.182704 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182705 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a773b93a-1df6-40c3-8bac-d2a8e7a4421a" containerName="cleanup" Oct 30 18:15:47.182815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182719 4603 cpu_manager.go:414] "containerMap: removing container" podUID="a773b93a-1df6-40c3-8bac-d2a8e7a4421a" containerName="cleanup" Oct 30 18:15:47.182815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182723 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a773b93a-1df6-40c3-8bac-d2a8e7a4421a" containerName="cleanup" Oct 30 18:15:47.182815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182735 4603 cpu_manager.go:414] "containerMap: removing container" podUID="8697996a-d257-4a4e-9c3c-e372ee1b0726" containerName="istio-proxy" Oct 30 18:15:47.182815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182741 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8697996a-d257-4a4e-9c3c-e372ee1b0726" containerName="istio-proxy" Oct 30 18:15:47.182815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182758 4603 cpu_manager.go:414] "containerMap: removing container" podUID="5fdac8a8-5038-402c-a3f2-c4393f772caa" containerName="cleanup" Oct 30 18:15:47.182815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182763 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fdac8a8-5038-402c-a3f2-c4393f772caa" containerName="cleanup" Oct 30 18:15:47.182815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182771 4603 cpu_manager.go:414] "containerMap: removing container" podUID="28e48ed8-4a07-45cb-ac86-959f20ae1c35" containerName="revoke-keys" Oct 30 18:15:47.182815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182776 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28e48ed8-4a07-45cb-ac86-959f20ae1c35" containerName="revoke-keys" Oct 30 18:15:47.182815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182789 4603 cpu_manager.go:414] "containerMap: removing container" podUID="c7e02b19-7f85-4e69-8a62-f5436875c9e4" containerName="istio-proxy" Oct 30 18:15:47.182815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182796 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7e02b19-7f85-4e69-8a62-f5436875c9e4" containerName="istio-proxy" Oct 30 18:15:47.182815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182806 4603 cpu_manager.go:414] "containerMap: removing container" podUID="44c65786-428f-4ac0-97db-020b7a909823" containerName="payload-pre-processing" Oct 30 18:15:47.182815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182811 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c65786-428f-4ac0-97db-020b7a909823" containerName="payload-pre-processing" Oct 30 18:15:47.182815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182818 4603 cpu_manager.go:414] "containerMap: removing container" podUID="61cec429-6d83-430a-9c51-2942c12dac9a" containerName="cleanup" Oct 30 18:15:47.183242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182822 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61cec429-6d83-430a-9c51-2942c12dac9a" containerName="cleanup" Oct 30 18:15:47.183242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182828 4603 cpu_manager.go:414] "containerMap: removing container" podUID="5fdac8a8-5038-402c-a3f2-c4393f772caa" containerName="cleanup" Oct 30 18:15:47.183242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182832 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fdac8a8-5038-402c-a3f2-c4393f772caa" containerName="cleanup" Oct 30 18:15:47.183242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182838 4603 cpu_manager.go:414] "containerMap: removing container" podUID="61cec429-6d83-430a-9c51-2942c12dac9a" containerName="cleanup" Oct 30 18:15:47.183242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182842 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61cec429-6d83-430a-9c51-2942c12dac9a" containerName="cleanup" Oct 30 18:15:47.183242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182850 4603 cpu_manager.go:414] "containerMap: removing container" podUID="c7e02b19-7f85-4e69-8a62-f5436875c9e4" containerName="istio-proxy" Oct 30 18:15:47.183242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182855 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7e02b19-7f85-4e69-8a62-f5436875c9e4" containerName="istio-proxy" Oct 30 18:15:47.183242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182867 4603 cpu_manager.go:414] "containerMap: removing container" podUID="61cec429-6d83-430a-9c51-2942c12dac9a" containerName="cleanup" Oct 30 18:15:47.183242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182871 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61cec429-6d83-430a-9c51-2942c12dac9a" containerName="cleanup" Oct 30 18:15:47.183242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182961 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="a773b93a-1df6-40c3-8bac-d2a8e7a4421a" containerName="cleanup" Oct 30 18:15:47.183242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182973 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="a773b93a-1df6-40c3-8bac-d2a8e7a4421a" containerName="cleanup" Oct 30 18:15:47.183242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182980 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="8697996a-d257-4a4e-9c3c-e372ee1b0726" containerName="istio-proxy" Oct 30 18:15:47.183242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182985 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fdac8a8-5038-402c-a3f2-c4393f772caa" containerName="cleanup" Oct 30 18:15:47.183242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182992 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="28e48ed8-4a07-45cb-ac86-959f20ae1c35" containerName="revoke-keys" Oct 30 18:15:47.183242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.182997 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7e02b19-7f85-4e69-8a62-f5436875c9e4" containerName="istio-proxy" Oct 30 18:15:47.183242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.183003 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c65786-428f-4ac0-97db-020b7a909823" containerName="payload-pre-processing" Oct 30 18:15:47.183242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.183009 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="61cec429-6d83-430a-9c51-2942c12dac9a" containerName="cleanup" Oct 30 18:15:47.183242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.183014 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fdac8a8-5038-402c-a3f2-c4393f772caa" containerName="cleanup" Oct 30 18:15:47.183242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.183018 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="61cec429-6d83-430a-9c51-2942c12dac9a" containerName="cleanup" Oct 30 18:15:47.183242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.183025 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7e02b19-7f85-4e69-8a62-f5436875c9e4" containerName="istio-proxy" Oct 30 18:15:47.183242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.183031 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="61cec429-6d83-430a-9c51-2942c12dac9a" containerName="cleanup" Oct 30 18:15:47.185534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.185514 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.200132 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.200113 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-da151d-openshift-default-dockercfg-wqbqs\"" Oct 30 18:15:47.216923 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.216898 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr"] Oct 30 18:15:47.263938 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.263868 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:47.350815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.347547 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-data\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.350815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.347623 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-workload-certs\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.350815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.347663 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-credential-socket\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.350815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.347692 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-workload-socket\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.350815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.347766 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istiod-ca-cert\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.350815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.347822 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj7bn\" (UniqueName: \"kubernetes.io/projected/a6ae148a-f57c-4430-b5ee-c17a7885bc78-kube-api-access-vj7bn\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.350815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.347846 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-podinfo\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.350815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.347879 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-token\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.350815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.348022 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-envoy\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.451731 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.451694 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-workload-certs\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.451992 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.451973 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-credential-socket\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.452106 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.452090 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-workload-socket\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.452275 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.452259 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istiod-ca-cert\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.466423 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.466390 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj7bn\" (UniqueName: \"kubernetes.io/projected/a6ae148a-f57c-4430-b5ee-c17a7885bc78-kube-api-access-vj7bn\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.466685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.466666 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-podinfo\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.466827 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.466812 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-token\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.467095 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.467082 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-envoy\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.467958 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.467935 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-data\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.468487 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.468465 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-data\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.469891 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.464353 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-credential-socket\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.470425 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.464569 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-workload-socket\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.513576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.496106 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-envoy\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.513576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.463432 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istiod-ca-cert\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.513576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.464155 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-workload-certs\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.513576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.500469 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-podinfo\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.546525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.538327 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:47.570537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.560693 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-token\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.570537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.563948 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj7bn\" (UniqueName: \"kubernetes.io/projected/a6ae148a-f57c-4430-b5ee-c17a7885bc78-kube-api-access-vj7bn\") pod \"e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:47.800666 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:47.800587 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:48.123586 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:15:48.123558 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ae148a_f57c_4430_b5ee_c17a7885bc78.slice/crio-87094005fe27f81cc08ff6e72d51f3a48b50eb46fe51713b8772a5fcff9200b1 WatchSource:0}: Error finding container 87094005fe27f81cc08ff6e72d51f3a48b50eb46fe51713b8772a5fcff9200b1: Status 404 returned error can't find the container with id 87094005fe27f81cc08ff6e72d51f3a48b50eb46fe51713b8772a5fcff9200b1 Oct 30 18:15:48.123586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:48.123576 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr"] Oct 30 18:15:48.125941 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:48.125913 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:48.126031 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:48.125971 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:48.126031 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:48.126001 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:48.533164 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:48.533125 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:48.818659 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:48.818569 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" event={"ID":"a6ae148a-f57c-4430-b5ee-c17a7885bc78","Type":"ContainerStarted","Data":"3030359edc4b632a8fd5a1e84a63364b37388b0be4da8fa8dc100d5a1fda6419"} Oct 30 18:15:48.819187 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:48.819164 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" event={"ID":"a6ae148a-f57c-4430-b5ee-c17a7885bc78","Type":"ContainerStarted","Data":"87094005fe27f81cc08ff6e72d51f3a48b50eb46fe51713b8772a5fcff9200b1"} Oct 30 18:15:48.911712 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:48.911648 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" podStartSLOduration=1.911626469 podStartE2EDuration="1.911626469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:48.894112707 +0000 UTC m=+1409.609682581" watchObservedRunningTime="2026-10-30 18:15:48.911626469 +0000 UTC m=+1409.627196344" Oct 30 18:15:49.083148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:49.083064 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:49.216372 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:49.216290 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl"] Oct 30 18:15:49.223906 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:49.223875 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0bdf03-m7spl"] Oct 30 18:15:49.248336 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:49.248301 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:49.397576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:49.397466 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:49.425091 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:49.425017 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28e48ed8-4a07-45cb-ac86-959f20ae1c35" path="/var/lib/kubelet/pods/28e48ed8-4a07-45cb-ac86-959f20ae1c35/volumes" Oct 30 18:15:49.786449 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:49.786418 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:49.801370 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:49.801349 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:49.806622 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:49.806603 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:49.824487 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:49.824470 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:49.825195 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:49.825178 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:15:51.071576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:51.071533 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:51.417667 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:51.417591 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:51.577350 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:51.577304 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:51.929744 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:15:51.929707 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:15:54.342105 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:54.342059 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:54.486529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:54.486466 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:54.627227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:54.626196 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:55.082528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:55.081047 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:15:56.377527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:56.372597 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:56.940583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:56.938123 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:57.140557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:57.128442 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:57.222586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:57.218461 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc"] Oct 30 18:15:57.222586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:57.218736 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" podUID="b4eac5a8-12c7-41b6-a8ac-8e570386586c" containerName="istio-proxy" containerID="cri-o://031e180f66ad5e401b3641040d346ab3515586a60ca1e676aceb66ff721e3769" gracePeriod=30 Oct 30 18:15:57.384547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:57.353264 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:57.779395 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:57.772995 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw"] Oct 30 18:15:57.791657 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:57.791617 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" podUID="59611c8e-86d7-44a9-b52f-a1a8ce2f0720" containerName="istio-proxy" containerID="cri-o://015a207e3260a204db163e81230625cdaaccc33fd343f6e14e5eb24fbb959c7f" gracePeriod=30 Oct 30 18:15:58.282641 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:58.282600 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:59.617691 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:15:59.617645 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.022587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:00.019789 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:16:00.304808 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:00.291591 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Oct 30 18:16:00.307663 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:00.307620 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.344998 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:00.344956 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:16:00.345673 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:00.345236 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:16:00.513423 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:00.509841 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.513423 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:00.510196 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:16:00.607580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:00.592623 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.638832 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:00.638775 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:16:00.767529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:00.756579 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:16:01.039696 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:01.039655 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:01.040611 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:01.040587 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:16:01.349582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:01.345832 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:01.448528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:01.387700 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:16:01.467297 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:01.439657 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:01.468604 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:01.468548 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:01.547534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:01.529461 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:01.605429 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:01.604474 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:01.617317 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:01.549099 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:01.756545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:01.756102 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:01.808844 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:01.796802 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:16:01.808844 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:01.796881 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:16:01.808844 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:01.796920 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:16:01.808844 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:01.796966 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:16:01.808844 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:01.797001 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:16:02.240156 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:02.240112 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:02.443569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:02.435736 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:16:02.469682 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:02.468692 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:02.469682 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:02.469399 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:02.525834 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:02.525805 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:02.661237 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:02.659892 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:16:02.676181 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:02.676140 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:16:02.677109 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:02.677078 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:16:02.759808 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:02.759765 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:03.364530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:03.344157 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:03.459650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:03.457815 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:03.475753 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:03.472610 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:16:03.484578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:03.478052 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:03.539687 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:03.539650 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:03.539950 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:03.539904 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" podUID="ec3c95aa-215c-4f15-8dc8-bc0beb954236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:03.698660 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:03.698616 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:03.787854 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:03.787812 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:16:03.839591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:03.806782 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:03.839591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:03.809408 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:03.977275 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:03.977229 4603 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 18:16:03.977574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:03.977540 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" podUID="59611c8e-86d7-44a9-b52f-a1a8ce2f0720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 18:16:04.121057 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:04.120955 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:04.121354 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:04.121318 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" podUID="187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:04.228034 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:04.227994 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:04.229266 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:04.229233 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" podUID="1d187b51-f5c6-4a63-8300-2f01184a15d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:04.280376 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:04.280339 4603 generic.go:356] "Generic (PLEG): container finished" podID="59611c8e-86d7-44a9-b52f-a1a8ce2f0720" containerID="015a207e3260a204db163e81230625cdaaccc33fd343f6e14e5eb24fbb959c7f" exitCode=0 Oct 30 18:16:04.280903 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:04.280577 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" event={"ID":"59611c8e-86d7-44a9-b52f-a1a8ce2f0720","Type":"ContainerDied","Data":"015a207e3260a204db163e81230625cdaaccc33fd343f6e14e5eb24fbb959c7f"} Oct 30 18:16:04.306840 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:04.306805 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:04.307127 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:04.307097 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" podUID="b6e43478-8cc1-4452-8bc8-fed14f127896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:04.462304 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:04.458774 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:16:04.462304 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:04.460055 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:16:04.520561 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:04.490748 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:16:04.530540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:04.498977 4603 generic.go:356] "Generic (PLEG): container finished" podID="b4eac5a8-12c7-41b6-a8ac-8e570386586c" containerID="031e180f66ad5e401b3641040d346ab3515586a60ca1e676aceb66ff721e3769" exitCode=0 Oct 30 18:16:04.530540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:04.499013 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" event={"ID":"b4eac5a8-12c7-41b6-a8ac-8e570386586c","Type":"ContainerDied","Data":"031e180f66ad5e401b3641040d346ab3515586a60ca1e676aceb66ff721e3769"} Oct 30 18:16:04.530540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:04.525166 4603 scope.go:122] "RemoveContainer" containerID="33046b182173b53bab6743bc3bfae8d29b429cb5333ea2642320c68b70a438cc" Oct 30 18:16:04.530540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:04.507235 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:04.686724 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:04.686679 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:04.731263 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:04.731158 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:04.845890 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:04.813618 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:04.845890 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:04.813992 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:16:04.845890 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:04.814254 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:16:05.490529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:05.483457 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:05.490529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:05.484672 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:16:05.564691 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:05.521477 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:05.564691 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:05.536126 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:05.564691 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:05.536599 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:05.784529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:05.781053 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:16:05.846529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:05.821873 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:16:05.884529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:05.827441 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:05.899909 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:05.827484 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:06.098001 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.097970 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:16:06.227168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.199572 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-workload-socket\") pod \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " Oct 30 18:16:06.227168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.199689 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-token\" (UniqueName: \"kubernetes.io/projected/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-token\") pod \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " Oct 30 18:16:06.227168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.199770 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-envoy\") pod \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " Oct 30 18:16:06.227168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.199857 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-data\") pod \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " Oct 30 18:16:06.227168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.199907 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-workload-certs\") pod \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " Oct 30 18:16:06.227168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.200002 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4eac5a8-12c7-41b6-a8ac-8e570386586c-kube-api-access-wcvv9\" (UniqueName: \"kubernetes.io/projected/b4eac5a8-12c7-41b6-a8ac-8e570386586c-kube-api-access-wcvv9\") pod \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " Oct 30 18:16:06.227168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.200069 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-podinfo\") pod \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " Oct 30 18:16:06.227168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.200098 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istiod-ca-cert\") pod \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " Oct 30 18:16:06.227168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.200186 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-credential-socket\") pod \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\" (UID: \"b4eac5a8-12c7-41b6-a8ac-8e570386586c\") " Oct 30 18:16:06.234693 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.234451 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-data" pod "b4eac5a8-12c7-41b6-a8ac-8e570386586c" (UID: "b4eac5a8-12c7-41b6-a8ac-8e570386586c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:16:06.235090 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.234895 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-credential-socket" pod "b4eac5a8-12c7-41b6-a8ac-8e570386586c" (UID: "b4eac5a8-12c7-41b6-a8ac-8e570386586c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:16:06.235518 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.235233 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-workload-socket" pod "b4eac5a8-12c7-41b6-a8ac-8e570386586c" (UID: "b4eac5a8-12c7-41b6-a8ac-8e570386586c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:16:06.235620 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.235589 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-workload-certs" pod "b4eac5a8-12c7-41b6-a8ac-8e570386586c" (UID: "b4eac5a8-12c7-41b6-a8ac-8e570386586c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:16:06.251528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.242402 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:16:06.251528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.243686 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istiod-ca-cert" pod "b4eac5a8-12c7-41b6-a8ac-8e570386586c" (UID: "b4eac5a8-12c7-41b6-a8ac-8e570386586c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:16:06.278603 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.278541 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-token" pod "b4eac5a8-12c7-41b6-a8ac-8e570386586c" (UID: "b4eac5a8-12c7-41b6-a8ac-8e570386586c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:06.296564 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.279713 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4eac5a8-12c7-41b6-a8ac-8e570386586c-kube-api-access-wcvv9" pod "b4eac5a8-12c7-41b6-a8ac-8e570386586c" (UID: "b4eac5a8-12c7-41b6-a8ac-8e570386586c"). InnerVolumeSpecName "kube-api-access-wcvv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:06.313525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.303016 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istiod-ca-cert\") pod \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " Oct 30 18:16:06.313525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.303087 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-workload-certs\") pod \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " Oct 30 18:16:06.313525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.303240 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-data\") pod \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " Oct 30 18:16:06.313525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.303302 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-workload-socket\") pod \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " Oct 30 18:16:06.313525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.303366 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-credential-socket\") pod \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " Oct 30 18:16:06.313525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.303476 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-envoy\") pod \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " Oct 30 18:16:06.313525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.303554 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-podinfo\") pod \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " Oct 30 18:16:06.313525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.303638 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-kube-api-access-4j25w\" (UniqueName: \"kubernetes.io/projected/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-kube-api-access-4j25w\") pod \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " Oct 30 18:16:06.313525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.303688 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-token\" (UniqueName: \"kubernetes.io/projected/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-token\") pod \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\" (UID: \"59611c8e-86d7-44a9-b52f-a1a8ce2f0720\") " Oct 30 18:16:06.313525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.304387 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.313525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.304412 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.313525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.304427 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcvv9\" (UniqueName: \"kubernetes.io/projected/b4eac5a8-12c7-41b6-a8ac-8e570386586c-kube-api-access-wcvv9\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.313525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.304443 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.313525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.304460 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.313525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.304474 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.313525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.304487 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.313525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.305170 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-workload-socket" pod "59611c8e-86d7-44a9-b52f-a1a8ce2f0720" (UID: "59611c8e-86d7-44a9-b52f-a1a8ce2f0720"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:16:06.313525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.305459 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istiod-ca-cert" pod "59611c8e-86d7-44a9-b52f-a1a8ce2f0720" (UID: "59611c8e-86d7-44a9-b52f-a1a8ce2f0720"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:16:06.313525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.305712 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-workload-certs" pod "59611c8e-86d7-44a9-b52f-a1a8ce2f0720" (UID: "59611c8e-86d7-44a9-b52f-a1a8ce2f0720"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:16:06.313525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.309994 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-data" pod "59611c8e-86d7-44a9-b52f-a1a8ce2f0720" (UID: "59611c8e-86d7-44a9-b52f-a1a8ce2f0720"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:16:06.314995 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.314971 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-credential-socket" pod "59611c8e-86d7-44a9-b52f-a1a8ce2f0720" (UID: "59611c8e-86d7-44a9-b52f-a1a8ce2f0720"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:16:06.342747 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.342690 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-envoy" pod "b4eac5a8-12c7-41b6-a8ac-8e570386586c" (UID: "b4eac5a8-12c7-41b6-a8ac-8e570386586c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:16:06.347961 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.347791 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-token" pod "59611c8e-86d7-44a9-b52f-a1a8ce2f0720" (UID: "59611c8e-86d7-44a9-b52f-a1a8ce2f0720"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:06.347961 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.347876 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-kube-api-access-4j25w" pod "59611c8e-86d7-44a9-b52f-a1a8ce2f0720" (UID: "59611c8e-86d7-44a9-b52f-a1a8ce2f0720"). InnerVolumeSpecName "kube-api-access-4j25w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:06.347961 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.347931 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-podinfo" pod "b4eac5a8-12c7-41b6-a8ac-8e570386586c" (UID: "b4eac5a8-12c7-41b6-a8ac-8e570386586c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:16:06.351043 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.350963 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-envoy" pod "59611c8e-86d7-44a9-b52f-a1a8ce2f0720" (UID: "59611c8e-86d7-44a9-b52f-a1a8ce2f0720"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:16:06.361670 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.361609 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-podinfo" pod "59611c8e-86d7-44a9-b52f-a1a8ce2f0720" (UID: "59611c8e-86d7-44a9-b52f-a1a8ce2f0720"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:16:06.424529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.417406 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j25w\" (UniqueName: \"kubernetes.io/projected/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-kube-api-access-4j25w\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.424529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.417440 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.424529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.417455 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.424529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.417467 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.424529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.417480 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.424529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.417493 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.424529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.417524 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.424529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.417537 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.424529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.417549 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.424529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.417561 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59611c8e-86d7-44a9-b52f-a1a8ce2f0720-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.424529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.417572 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4eac5a8-12c7-41b6-a8ac-8e570386586c-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.511559 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.510850 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:06.548530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.511770 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:06.714740 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.714697 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:16:06.725542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.717035 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:16:06.787574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.787530 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:06.887529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.886605 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:16:06.909556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.900872 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:06.909556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:06.901245 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:16:07.124263 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:07.124212 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:07.192539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:07.172380 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" event={"ID":"b4eac5a8-12c7-41b6-a8ac-8e570386586c","Type":"ContainerDied","Data":"aa91ae0f9cecd80892404bf1b500db60541a38551ff1d8a28e6bef25510ff197"} Oct 30 18:16:07.192539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:07.172457 4603 scope.go:122] "RemoveContainer" containerID="031e180f66ad5e401b3641040d346ab3515586a60ca1e676aceb66ff721e3769" Oct 30 18:16:07.192539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:07.172634 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc" Oct 30 18:16:07.291558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:07.277944 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" event={"ID":"59611c8e-86d7-44a9-b52f-a1a8ce2f0720","Type":"ContainerDied","Data":"610c2f8d8eb17c61d111dc9c74e006f1cf00393c2ab5076ec5e3b81683cbd750"} Oct 30 18:16:07.291558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:07.278098 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw" Oct 30 18:16:07.365536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:07.350189 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:07.394588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:07.349662 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:07.557556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:07.531987 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:07.783186 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:07.648004 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:16:07.783897 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:07.667763 4603 scope.go:122] "RemoveContainer" containerID="015a207e3260a204db163e81230625cdaaccc33fd343f6e14e5eb24fbb959c7f" Oct 30 18:16:07.784152 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:07.757568 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:07.901958 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:07.801174 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:07.901958 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:07.868582 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:07.926697 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:07.926648 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:07.927412 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:07.927379 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:07.931862 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:07.931833 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:16:08.109647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:08.109594 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:08.513631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:08.507139 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 30 18:16:08.513631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:08.507731 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw"] Oct 30 18:16:08.590540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:08.587710 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-dqvjw"] Oct 30 18:16:08.674779 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:08.674677 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:08.702671 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:08.702596 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc"] Oct 30 18:16:08.813531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:08.792687 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:08.813531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:08.800574 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:16:08.813531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:08.800962 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:16:08.813531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:08.802395 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:08.830728 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:08.830655 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-ptrfc"] Oct 30 18:16:08.870433 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:08.868879 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:08.941742 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:08.941638 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:08.942726 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:08.942692 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:16:08.943184 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:08.943160 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:16:09.041535 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:16:09.029146 4603 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59611c8e_86d7_44a9_b52f_a1a8ce2f0720.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:16:09.164524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:09.164440 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:09.857528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:09.848198 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:09.857528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:09.848621 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:09.857528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:09.848932 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:16:09.876527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:09.873641 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:09.904528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:09.903645 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:09.962582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:09.946567 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:09.966528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:09.946627 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:16:09.974526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:09.946712 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:10.164725 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.164638 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59611c8e-86d7-44a9-b52f-a1a8ce2f0720" path="/var/lib/kubelet/pods/59611c8e-86d7-44a9-b52f-a1a8ce2f0720/volumes" Oct 30 18:16:10.191567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.185856 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4eac5a8-12c7-41b6-a8ac-8e570386586c" path="/var/lib/kubelet/pods/b4eac5a8-12c7-41b6-a8ac-8e570386586c/volumes" Oct 30 18:16:10.231430 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.231402 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:16:10.252577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.252551 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:16:10.314419 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.309257 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:16:10.314419 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.310346 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:16:10.321398 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.320721 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:16:10.331645 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.330945 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:16:10.332240 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.332223 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:16:10.332835 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.332811 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:16:10.741067 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.741031 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf"] Oct 30 18:16:10.741659 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.741643 4603 cpu_manager.go:414] "containerMap: removing container" podUID="59611c8e-86d7-44a9-b52f-a1a8ce2f0720" containerName="istio-proxy" Oct 30 18:16:10.741710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.741661 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59611c8e-86d7-44a9-b52f-a1a8ce2f0720" containerName="istio-proxy" Oct 30 18:16:10.741710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.741671 4603 cpu_manager.go:414] "containerMap: removing container" podUID="a773b93a-1df6-40c3-8bac-d2a8e7a4421a" containerName="cleanup" Oct 30 18:16:10.741710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.741676 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a773b93a-1df6-40c3-8bac-d2a8e7a4421a" containerName="cleanup" Oct 30 18:16:10.741710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.741693 4603 cpu_manager.go:414] "containerMap: removing container" podUID="b4eac5a8-12c7-41b6-a8ac-8e570386586c" containerName="istio-proxy" Oct 30 18:16:10.741710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.741699 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4eac5a8-12c7-41b6-a8ac-8e570386586c" containerName="istio-proxy" Oct 30 18:16:10.741866 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.741812 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4eac5a8-12c7-41b6-a8ac-8e570386586c" containerName="istio-proxy" Oct 30 18:16:10.741866 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.741824 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="a773b93a-1df6-40c3-8bac-d2a8e7a4421a" containerName="cleanup" Oct 30 18:16:10.741866 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.741831 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4eac5a8-12c7-41b6-a8ac-8e570386586c" containerName="istio-proxy" Oct 30 18:16:10.741866 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.741840 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="59611c8e-86d7-44a9-b52f-a1a8ce2f0720" containerName="istio-proxy" Oct 30 18:16:10.753734 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.753697 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" Oct 30 18:16:10.768360 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.768329 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf"] Oct 30 18:16:10.774294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.774267 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-da151d-serving-cert\"" Oct 30 18:16:10.841075 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.841045 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-openshift-service-ca\") pod \"maas-api-e2e-api-b-da151d-6b88bc8445-q92zf\" (UID: \"5a6d0fd2-c58a-4df1-95c2-754554fd69d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" Oct 30 18:16:10.841280 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.841095 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-maas-api-tls\") pod \"maas-api-e2e-api-b-da151d-6b88bc8445-q92zf\" (UID: \"5a6d0fd2-c58a-4df1-95c2-754554fd69d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" Oct 30 18:16:10.841280 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.841264 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltz7r\" (UniqueName: \"kubernetes.io/projected/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-kube-api-access-ltz7r\") pod \"maas-api-e2e-api-b-da151d-6b88bc8445-q92zf\" (UID: \"5a6d0fd2-c58a-4df1-95c2-754554fd69d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" Oct 30 18:16:10.841533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.841497 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-metrics-certs\") pod \"maas-api-e2e-api-b-da151d-6b88bc8445-q92zf\" (UID: \"5a6d0fd2-c58a-4df1-95c2-754554fd69d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" Oct 30 18:16:10.852355 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.852325 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:10.949581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.949527 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-openshift-service-ca\") pod \"maas-api-e2e-api-b-da151d-6b88bc8445-q92zf\" (UID: \"5a6d0fd2-c58a-4df1-95c2-754554fd69d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" Oct 30 18:16:10.949888 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.949866 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-maas-api-tls\") pod \"maas-api-e2e-api-b-da151d-6b88bc8445-q92zf\" (UID: \"5a6d0fd2-c58a-4df1-95c2-754554fd69d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" Oct 30 18:16:10.950124 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.950106 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltz7r\" (UniqueName: \"kubernetes.io/projected/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-kube-api-access-ltz7r\") pod \"maas-api-e2e-api-b-da151d-6b88bc8445-q92zf\" (UID: \"5a6d0fd2-c58a-4df1-95c2-754554fd69d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" Oct 30 18:16:10.966005 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.965967 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-metrics-certs\") pod \"maas-api-e2e-api-b-da151d-6b88bc8445-q92zf\" (UID: \"5a6d0fd2-c58a-4df1-95c2-754554fd69d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" Oct 30 18:16:10.972046 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.957612 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-openshift-service-ca\") pod \"maas-api-e2e-api-b-da151d-6b88bc8445-q92zf\" (UID: \"5a6d0fd2-c58a-4df1-95c2-754554fd69d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" Oct 30 18:16:11.000583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:10.995653 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-metrics-certs\") pod \"maas-api-e2e-api-b-da151d-6b88bc8445-q92zf\" (UID: \"5a6d0fd2-c58a-4df1-95c2-754554fd69d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" Oct 30 18:16:11.025529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:11.020593 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-maas-api-tls\") pod \"maas-api-e2e-api-b-da151d-6b88bc8445-q92zf\" (UID: \"5a6d0fd2-c58a-4df1-95c2-754554fd69d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" Oct 30 18:16:11.025529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:11.024590 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltz7r\" (UniqueName: \"kubernetes.io/projected/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-kube-api-access-ltz7r\") pod \"maas-api-e2e-api-b-da151d-6b88bc8445-q92zf\" (UID: \"5a6d0fd2-c58a-4df1-95c2-754554fd69d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" Oct 30 18:16:11.029528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:11.028024 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:11.092513 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:11.085831 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" Oct 30 18:16:11.475645 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:11.475618 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf"] Oct 30 18:16:11.477537 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:16:11.477495 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6d0fd2_c58a_4df1_95c2_754554fd69d3.slice/crio-cdfd74d793b084b563a08861b6d6dd54dc7b76701ac3ffb40a9b31537d254557 WatchSource:0}: Error finding container cdfd74d793b084b563a08861b6d6dd54dc7b76701ac3ffb40a9b31537d254557: Status 404 returned error can't find the container with id cdfd74d793b084b563a08861b6d6dd54dc7b76701ac3ffb40a9b31537d254557 Oct 30 18:16:11.479655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:11.479630 4603 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:16:11.878372 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:11.878344 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" event={"ID":"5a6d0fd2-c58a-4df1-95c2-754554fd69d3","Type":"ContainerStarted","Data":"cdfd74d793b084b563a08861b6d6dd54dc7b76701ac3ffb40a9b31537d254557"} Oct 30 18:16:12.726647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:12.726617 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:12.887409 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:12.887327 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" event={"ID":"5a6d0fd2-c58a-4df1-95c2-754554fd69d3","Type":"ContainerStarted","Data":"0a16a67b1b52599e53682472f8c9c078fb7346501d5eeecd2e06d394eba9ebcb"} Oct 30 18:16:12.887571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:12.887416 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" Oct 30 18:16:12.931599 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:12.931547 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" podStartSLOduration=2.565395207 podStartE2EDuration="2.931528936s" totalImagesPullingTime="366.133729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:12.927918218 +0000 UTC m=+1433.643488094" watchObservedRunningTime="2026-10-30 18:16:12.931528936 +0000 UTC m=+1433.647098809" Oct 30 18:16:14.423589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:14.423548 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:14.571686 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:14.571071 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:16.033285 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:16.033246 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:16.916366 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:16.916298 4603 generic.go:356] "Generic (PLEG): container finished" podID="3bb952b3-3096-442b-8ab6-2909cee2c021" containerID="29f2db7a9fed86224bd7bbf5685d606ede4d5b3a0c5a6fff0f036271f11bff75" exitCode=0 Oct 30 18:16:16.916551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:16.916367 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" event={"ID":"3bb952b3-3096-442b-8ab6-2909cee2c021","Type":"ContainerDied","Data":"29f2db7a9fed86224bd7bbf5685d606ede4d5b3a0c5a6fff0f036271f11bff75"} Oct 30 18:16:16.916775 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:16.916760 4603 scope.go:122] "RemoveContainer" containerID="29f2db7a9fed86224bd7bbf5685d606ede4d5b3a0c5a6fff0f036271f11bff75" Oct 30 18:16:16.918177 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:16.918151 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:16.918242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:16.918221 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:16.918277 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:16.918258 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:17.747844 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:17.747809 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:17.934259 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:17.934222 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" event={"ID":"3bb952b3-3096-442b-8ab6-2909cee2c021","Type":"ContainerStarted","Data":"cd0c93c47c474940e8505e75028c6d6d4340adfa764f47dcd83af08dde324fda"} Oct 30 18:16:18.711869 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:18.711832 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:16:18.712228 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:18.712139 4603 patch_prober.go:29] interesting pod/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:16:18.712405 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:18.712379 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" podUID="3bb952b3-3096-442b-8ab6-2909cee2c021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 18:16:18.917971 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:18.917943 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" Oct 30 18:16:19.427152 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:19.427123 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:19.713218 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:19.713003 4603 patch_prober.go:29] interesting pod/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:16:19.713218 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:19.713106 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" podUID="3bb952b3-3096-442b-8ab6-2909cee2c021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 18:16:20.713869 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:20.713830 4603 patch_prober.go:29] interesting pod/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:16:20.714369 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:20.713899 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" podUID="3bb952b3-3096-442b-8ab6-2909cee2c021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 18:16:21.132539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:21.132486 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:21.713914 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:21.713876 4603 patch_prober.go:29] interesting pod/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:16:21.714322 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:21.713942 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" podUID="3bb952b3-3096-442b-8ab6-2909cee2c021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 18:16:21.941936 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:16:21.941899 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:16:22.714623 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:22.714585 4603 patch_prober.go:29] interesting pod/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:16:22.715109 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:22.714659 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" podUID="3bb952b3-3096-442b-8ab6-2909cee2c021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 18:16:22.741954 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:22.741922 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:23.715631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:23.715602 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:16:23.716027 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:23.715999 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:16:23.716601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:23.716581 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:16:24.511652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:24.511620 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:26.144098 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:26.144064 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:27.916555 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:27.915173 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:29.635423 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:29.635377 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:31.271596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:31.271564 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:32.848296 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:32.848259 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:34.547267 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:34.547235 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:36.155064 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:36.155026 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:37.883524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:37.879708 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:39.436970 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:39.436937 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:40.995975 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:40.995924 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:42.068547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:42.068491 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:42.728525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:42.725467 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:44.281339 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:44.281287 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:44.744063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:44.743994 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7325-632f59"] Oct 30 18:16:44.744590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:44.744576 4603 cpu_manager.go:414] "containerMap: removing container" podUID="b4eac5a8-12c7-41b6-a8ac-8e570386586c" containerName="istio-proxy" Oct 30 18:16:44.744647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:44.744592 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4eac5a8-12c7-41b6-a8ac-8e570386586c" containerName="istio-proxy" Oct 30 18:16:44.746851 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:44.746836 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-632f59" Oct 30 18:16:44.762019 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:44.761995 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 18:16:44.764877 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:44.764857 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7325-632f59"] Oct 30 18:16:44.773219 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:44.773201 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 18:16:44.773219 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:44.773205 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vdx7j\"" Oct 30 18:16:44.831246 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:44.831214 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsdsq\" (UniqueName: \"kubernetes.io/projected/3f2d039a-5130-491e-81cd-fcc9881b4bd8-kube-api-access-lsdsq\") pod \"test-curl-7325-632f59\" (UID: \"3f2d039a-5130-491e-81cd-fcc9881b4bd8\") " pod="opendatahub/test-curl-7325-632f59" Oct 30 18:16:44.932850 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:44.932814 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsdsq\" (UniqueName: \"kubernetes.io/projected/3f2d039a-5130-491e-81cd-fcc9881b4bd8-kube-api-access-lsdsq\") pod \"test-curl-7325-632f59\" (UID: \"3f2d039a-5130-491e-81cd-fcc9881b4bd8\") " pod="opendatahub/test-curl-7325-632f59" Oct 30 18:16:44.955586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:44.955566 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsdsq\" (UniqueName: \"kubernetes.io/projected/3f2d039a-5130-491e-81cd-fcc9881b4bd8-kube-api-access-lsdsq\") pod \"test-curl-7325-632f59\" (UID: \"3f2d039a-5130-491e-81cd-fcc9881b4bd8\") " pod="opendatahub/test-curl-7325-632f59" Oct 30 18:16:45.057937 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:45.057905 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-632f59" Oct 30 18:16:45.365089 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:45.365057 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7325-632f59"] Oct 30 18:16:45.365541 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:16:45.365461 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f2d039a_5130_491e_81cd_fcc9881b4bd8.slice/crio-7fa69f08996e6a013975fafea6ced4432380dc4b56e927f2d8a225de647f4138 WatchSource:0}: Error finding container 7fa69f08996e6a013975fafea6ced4432380dc4b56e927f2d8a225de647f4138: Status 404 returned error can't find the container with id 7fa69f08996e6a013975fafea6ced4432380dc4b56e927f2d8a225de647f4138 Oct 30 18:16:45.846149 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:45.846108 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:46.181607 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:46.181530 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7325-632f59" event={"ID":"3f2d039a-5130-491e-81cd-fcc9881b4bd8","Type":"ContainerStarted","Data":"7fa69f08996e6a013975fafea6ced4432380dc4b56e927f2d8a225de647f4138"} Oct 30 18:16:47.523422 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:47.523394 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:48.196430 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:48.196394 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7325-632f59" event={"ID":"3f2d039a-5130-491e-81cd-fcc9881b4bd8","Type":"ContainerStarted","Data":"c01a14d38e60140a13340ea89c3a03bbe14b1fcc5baa1dbbc0bdd3b84d38d755"} Oct 30 18:16:48.235722 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:48.235674 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7325-632f59" podStartSLOduration=2.2682066 podStartE2EDuration="4.235659237s" totalImagesPullingTime="1.967452637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:48.233668006 +0000 UTC m=+1468.949237880" watchObservedRunningTime="2026-10-30 18:16:48.235659237 +0000 UTC m=+1468.951229111" Oct 30 18:16:49.130615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:49.130583 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7325-632f59"] Oct 30 18:16:49.135647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:49.135622 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7325-632f59"] Oct 30 18:16:49.292495 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:49.292416 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:50.305398 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:50.305359 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7325-632f59" podUID="3f2d039a-5130-491e-81cd-fcc9881b4bd8" containerName="test-curl-7325-632f59" containerID="cri-o://c01a14d38e60140a13340ea89c3a03bbe14b1fcc5baa1dbbc0bdd3b84d38d755" gracePeriod=2 Oct 30 18:16:50.964258 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:50.964226 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:51.929384 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:16:51.929350 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:16:52.512714 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:52.512692 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-632f59" Oct 30 18:16:52.550172 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:52.550150 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f2d039a-5130-491e-81cd-fcc9881b4bd8-kube-api-access-lsdsq\" (UniqueName: \"kubernetes.io/projected/3f2d039a-5130-491e-81cd-fcc9881b4bd8-kube-api-access-lsdsq\") pod \"3f2d039a-5130-491e-81cd-fcc9881b4bd8\" (UID: \"3f2d039a-5130-491e-81cd-fcc9881b4bd8\") " Oct 30 18:16:52.552543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:52.552517 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2d039a-5130-491e-81cd-fcc9881b4bd8-kube-api-access-lsdsq" pod "3f2d039a-5130-491e-81cd-fcc9881b4bd8" (UID: "3f2d039a-5130-491e-81cd-fcc9881b4bd8"). InnerVolumeSpecName "kube-api-access-lsdsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:52.645942 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:52.645916 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:52.675280 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:52.675237 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsdsq\" (UniqueName: \"kubernetes.io/projected/3f2d039a-5130-491e-81cd-fcc9881b4bd8-kube-api-access-lsdsq\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:16:53.324390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:53.324351 4603 generic.go:356] "Generic (PLEG): container finished" podID="3f2d039a-5130-491e-81cd-fcc9881b4bd8" containerID="c01a14d38e60140a13340ea89c3a03bbe14b1fcc5baa1dbbc0bdd3b84d38d755" exitCode=137 Oct 30 18:16:53.324943 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:53.324413 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-632f59" Oct 30 18:16:53.324943 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:53.324458 4603 scope.go:122] "RemoveContainer" containerID="c01a14d38e60140a13340ea89c3a03bbe14b1fcc5baa1dbbc0bdd3b84d38d755" Oct 30 18:16:53.334677 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:53.334658 4603 scope.go:122] "RemoveContainer" containerID="c01a14d38e60140a13340ea89c3a03bbe14b1fcc5baa1dbbc0bdd3b84d38d755" Oct 30 18:16:53.334952 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:16:53.334932 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c01a14d38e60140a13340ea89c3a03bbe14b1fcc5baa1dbbc0bdd3b84d38d755\": container with ID starting with c01a14d38e60140a13340ea89c3a03bbe14b1fcc5baa1dbbc0bdd3b84d38d755 not found: ID does not exist" containerID="c01a14d38e60140a13340ea89c3a03bbe14b1fcc5baa1dbbc0bdd3b84d38d755" Oct 30 18:16:53.335027 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:53.334960 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c01a14d38e60140a13340ea89c3a03bbe14b1fcc5baa1dbbc0bdd3b84d38d755"} err="failed to get container status \"c01a14d38e60140a13340ea89c3a03bbe14b1fcc5baa1dbbc0bdd3b84d38d755\": rpc error: code = NotFound desc = could not find container \"c01a14d38e60140a13340ea89c3a03bbe14b1fcc5baa1dbbc0bdd3b84d38d755\": container with ID starting with c01a14d38e60140a13340ea89c3a03bbe14b1fcc5baa1dbbc0bdd3b84d38d755 not found: ID does not exist" Oct 30 18:16:53.345976 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:16:53.345950 4603 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7325-632f59\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="3f2d039a-5130-491e-81cd-fcc9881b4bd8" pod="opendatahub/test-curl-7325-632f59" Oct 30 18:16:53.411058 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:53.411029 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f2d039a-5130-491e-81cd-fcc9881b4bd8" path="/var/lib/kubelet/pods/3f2d039a-5130-491e-81cd-fcc9881b4bd8/volumes" Oct 30 18:16:54.441880 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:54.441844 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:56.257561 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:56.257523 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:57.846631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:57.846595 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:59.568528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:16:59.559668 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:00.436966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:00.436924 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:00.601533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:00.600768 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:01.198542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:01.192512 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:02.791989 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:02.791909 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:04.513547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:04.512849 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:04.794429 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:04.794399 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p"] Oct 30 18:17:04.935978 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:04.935939 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:05.120436 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:05.120328 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:06.245979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:06.245942 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:07.853266 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:07.853219 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:07.942534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:07.938617 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:17:09.640690 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:09.640654 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:09.802527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:09.800578 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p" podUID="9d600055-7b4d-4586-9ca5-1305adb1983c" containerName="payload-pre-processing" containerID="cri-o://9f2d0808c67aa843b9cf6e0b3906cd69b578d712a93a33e0b6f04e834ca956f7" gracePeriod=25 Oct 30 18:17:10.176125 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:10.176069 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p" Oct 30 18:17:10.294184 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:10.294152 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d600055-7b4d-4586-9ca5-1305adb1983c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d600055-7b4d-4586-9ca5-1305adb1983c-plugins-config-volume\") pod \"9d600055-7b4d-4586-9ca5-1305adb1983c\" (UID: \"9d600055-7b4d-4586-9ca5-1305adb1983c\") " Oct 30 18:17:10.294387 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:10.294372 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d600055-7b4d-4586-9ca5-1305adb1983c-kube-api-access-cnjd5\" (UniqueName: \"kubernetes.io/projected/9d600055-7b4d-4586-9ca5-1305adb1983c-kube-api-access-cnjd5\") pod \"9d600055-7b4d-4586-9ca5-1305adb1983c\" (UID: \"9d600055-7b4d-4586-9ca5-1305adb1983c\") " Oct 30 18:17:10.294547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:10.294494 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d600055-7b4d-4586-9ca5-1305adb1983c-plugins-config-volume" pod "9d600055-7b4d-4586-9ca5-1305adb1983c" (UID: "9d600055-7b4d-4586-9ca5-1305adb1983c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:10.294828 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:10.294813 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d600055-7b4d-4586-9ca5-1305adb1983c-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:17:10.296739 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:10.296711 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d600055-7b4d-4586-9ca5-1305adb1983c-kube-api-access-cnjd5" pod "9d600055-7b4d-4586-9ca5-1305adb1983c" (UID: "9d600055-7b4d-4586-9ca5-1305adb1983c"). InnerVolumeSpecName "kube-api-access-cnjd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:10.404381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:10.400453 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnjd5\" (UniqueName: \"kubernetes.io/projected/9d600055-7b4d-4586-9ca5-1305adb1983c-kube-api-access-cnjd5\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:17:10.669066 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:10.668620 4603 generic.go:356] "Generic (PLEG): container finished" podID="9d600055-7b4d-4586-9ca5-1305adb1983c" containerID="9f2d0808c67aa843b9cf6e0b3906cd69b578d712a93a33e0b6f04e834ca956f7" exitCode=0 Oct 30 18:17:10.669066 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:10.668807 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p" event={"ID":"9d600055-7b4d-4586-9ca5-1305adb1983c","Type":"ContainerDied","Data":"9f2d0808c67aa843b9cf6e0b3906cd69b578d712a93a33e0b6f04e834ca956f7"} Oct 30 18:17:10.669066 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:10.668880 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p" event={"ID":"9d600055-7b4d-4586-9ca5-1305adb1983c","Type":"ContainerDied","Data":"77dfe2b1a553c617d772714af2f62f01844b5891004473cafa84e0116a92149e"} Oct 30 18:17:10.669066 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:10.668887 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p" Oct 30 18:17:10.669066 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:10.668914 4603 scope.go:122] "RemoveContainer" containerID="9f2d0808c67aa843b9cf6e0b3906cd69b578d712a93a33e0b6f04e834ca956f7" Oct 30 18:17:10.727967 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:10.727942 4603 scope.go:122] "RemoveContainer" containerID="9f2d0808c67aa843b9cf6e0b3906cd69b578d712a93a33e0b6f04e834ca956f7" Oct 30 18:17:10.732068 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:17:10.732042 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f2d0808c67aa843b9cf6e0b3906cd69b578d712a93a33e0b6f04e834ca956f7\": container with ID starting with 9f2d0808c67aa843b9cf6e0b3906cd69b578d712a93a33e0b6f04e834ca956f7 not found: ID does not exist" containerID="9f2d0808c67aa843b9cf6e0b3906cd69b578d712a93a33e0b6f04e834ca956f7" Oct 30 18:17:10.732261 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:10.732236 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f2d0808c67aa843b9cf6e0b3906cd69b578d712a93a33e0b6f04e834ca956f7"} err="failed to get container status \"9f2d0808c67aa843b9cf6e0b3906cd69b578d712a93a33e0b6f04e834ca956f7\": rpc error: code = NotFound desc = could not find container \"9f2d0808c67aa843b9cf6e0b3906cd69b578d712a93a33e0b6f04e834ca956f7\": container with ID starting with 9f2d0808c67aa843b9cf6e0b3906cd69b578d712a93a33e0b6f04e834ca956f7 not found: ID does not exist" Oct 30 18:17:10.783547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:10.783488 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p"] Oct 30 18:17:10.796579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:10.795906 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p"] Oct 30 18:17:11.094530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:11.093691 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c8f010-95b94dbdc-5885p" podUID="9d600055-7b4d-4586-9ca5-1305adb1983c" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.89:50052\" within 1s: context deadline exceeded" Oct 30 18:17:11.094530 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:17:11.093931 4603 logging.go:55] [core] [Channel #647 SubChannel #648]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.89:50052", ServerName: "10.133.0.89:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.89:50052: operation was canceled" Oct 30 18:17:11.447577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:11.442565 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d600055-7b4d-4586-9ca5-1305adb1983c" path="/var/lib/kubelet/pods/9d600055-7b4d-4586-9ca5-1305adb1983c/volumes" Oct 30 18:17:11.660532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:11.650034 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:12.468540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:12.466752 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:12.468540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:12.467128 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:17:12.472537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:12.469900 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:12.486533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:12.485572 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:12.711532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:12.706613 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:12.711532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:12.707857 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:12.814530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:12.813856 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:12.835586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:12.818571 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:17:13.373526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:13.372614 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:17:13.373526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:13.372684 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" podUID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:13.480528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:13.473600 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:13.480528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:13.474055 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:17:13.480528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:13.474327 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:17:13.480528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:13.474927 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:17:13.480528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:13.474984 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:17:13.480528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:13.475022 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:17:13.490547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:13.486632 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:13.490547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:13.487162 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:17:13.773526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:13.770869 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:17:13.808900 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:13.808856 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:17:13.844527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:13.830254 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:17:13.957551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:13.856066 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:17:13.957551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:13.881567 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:17:13.957551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:13.881889 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:17:13.957551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:13.948814 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:17:13.957551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:13.948904 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:17:14.475753 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:14.475643 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:14.484578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:14.476648 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:17:14.492574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:14.476719 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:17:14.553686 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:14.553629 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:14.675964 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:14.675918 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:14.846527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:14.836881 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:17:14.915530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:14.859595 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:17:14.947191 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:14.941070 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:14.947977 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:14.947941 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:17:15.125694 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:15.122208 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:15.500902 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:15.500795 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:17:15.504528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:15.502269 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:17:15.515353 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:15.514832 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:15.681638 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:15.681590 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Oct 30 18:17:15.705678 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:15.705628 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:16.021461 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:16.021417 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:16.031531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:16.024810 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:17:16.031531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:16.025233 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:16.032199 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:16.031867 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:17:16.510537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:16.510412 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:16.516488 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:16.516454 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:17:16.517618 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:16.517591 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:16.707651 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:16.707599 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:17.022870 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:17.022820 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:17.027839 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:17.027803 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:17.028442 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:17.028125 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:17.048685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:17.048633 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:17:17.120993 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:17.120948 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:17.341723 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:17.341617 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:17.374837 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:17.374803 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:17.515712 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:17.515653 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:17.517588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:17.517542 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:17.519478 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:17.519451 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:17:17.714810 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:17.714684 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:17:18.032528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:18.029608 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:18.032528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:18.031667 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:18.033533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:18.033443 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:17:18.057963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:18.056582 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:17:18.082855 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:18.082595 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:18.433682 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:18.433584 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:18.524616 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:18.523591 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:17:18.525213 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:18.525185 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:17:18.525811 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:18.525361 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:18.607646 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:18.607599 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:18.729016 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:18.727984 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:18.954533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:18.950608 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:17:18.954533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:18.950697 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" podUID="b664f4ca-736e-4cb1-991d-985d1e6aff34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:19.002801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:19.002697 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:17:19.003094 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:19.003048 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" podUID="4d6cd466-9f15-476d-9f04-6d03176f025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:19.030173 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:19.030121 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:19.034754 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:19.034681 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:19.035698 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:19.035657 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:19.071529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:19.068603 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:19.180773 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:19.180285 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:17:19.180773 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:19.180345 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" podUID="d4a3adf6-fb23-4334-8bab-f800b4e8e089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:19.597652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:19.597566 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:17:19.848603 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:19.819092 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:17:19.848603 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:19.819465 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:17:19.848603 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:19.833264 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:17:20.032909 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:20.030576 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:20.033233 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:20.033207 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:20.039909 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:20.039862 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:17:20.047745 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:20.045606 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:20.079097 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:20.077673 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:20.302724 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:20.302679 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:20.534237 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:20.534209 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 18:17:20.535825 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:20.535804 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 18:17:20.618576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:20.616558 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:17:20.826528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:20.823593 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:20.826528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:20.824773 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:20.839304 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:20.834620 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:20.882856 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:20.882773 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:17:21.079972 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:21.079927 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:21.081244 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:21.081198 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:21.128528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:21.094099 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:21.574913 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:21.574874 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:21.626530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:21.617251 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:17:21.825732 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:21.825628 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:21.826357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:21.826326 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:21.855123 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:21.855072 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:17:21.909145 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:21.909115 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:17:22.059901 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:22.059852 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:22.218583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:22.085888 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:17:22.259082 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:22.085903 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:22.294564 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:22.110714 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:22.302835 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:17:22.176328 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:17:22.439525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:22.435901 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:17:22.616947 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:22.606797 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:17:22.623438 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:22.623398 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:22.731759 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:22.731723 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:17:22.739058 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:22.739039 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:17:22.739949 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:22.739928 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:17:22.740560 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:22.740539 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:17:23.243732 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:23.243693 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:23.812527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:23.812017 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:23.951510 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:23.951473 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:24.741469 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:24.741440 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:26.452529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:26.451218 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:27.763545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:27.763494 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:28.228174 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:28.228142 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:28.507279 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:28.507016 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:28.663142 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:28.663108 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:28.838228 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:28.837402 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:28.997658 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:28.997632 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:29.153969 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:29.153791 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:29.575696 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:29.575658 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:29.737897 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:29.737864 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:29.896186 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:29.896109 4603 generic.go:356] "Generic (PLEG): container finished" podID="a6ae148a-f57c-4430-b5ee-c17a7885bc78" containerID="3030359edc4b632a8fd5a1e84a63364b37388b0be4da8fa8dc100d5a1fda6419" exitCode=0 Oct 30 18:17:29.896641 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:29.896184 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" event={"ID":"a6ae148a-f57c-4430-b5ee-c17a7885bc78","Type":"ContainerDied","Data":"3030359edc4b632a8fd5a1e84a63364b37388b0be4da8fa8dc100d5a1fda6419"} Oct 30 18:17:29.896726 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:29.896653 4603 scope.go:122] "RemoveContainer" containerID="3030359edc4b632a8fd5a1e84a63364b37388b0be4da8fa8dc100d5a1fda6419" Oct 30 18:17:29.898092 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:29.898067 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:17:29.898154 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:29.898133 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:17:29.898196 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:29.898165 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:17:30.222416 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.222331 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl"] Oct 30 18:17:30.223023 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.223002 4603 cpu_manager.go:414] "containerMap: removing container" podUID="9d600055-7b4d-4586-9ca5-1305adb1983c" containerName="payload-pre-processing" Oct 30 18:17:30.223124 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.223025 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d600055-7b4d-4586-9ca5-1305adb1983c" containerName="payload-pre-processing" Oct 30 18:17:30.223124 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.223078 4603 cpu_manager.go:414] "containerMap: removing container" podUID="3f2d039a-5130-491e-81cd-fcc9881b4bd8" containerName="test-curl-7325-632f59" Oct 30 18:17:30.223124 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.223086 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f2d039a-5130-491e-81cd-fcc9881b4bd8" containerName="test-curl-7325-632f59" Oct 30 18:17:30.223285 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.223253 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f2d039a-5130-491e-81cd-fcc9881b4bd8" containerName="test-curl-7325-632f59" Oct 30 18:17:30.223285 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.223273 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d600055-7b4d-4586-9ca5-1305adb1983c" containerName="payload-pre-processing" Oct 30 18:17:30.225748 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.225727 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.240467 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.240437 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-853d3a-openshift-default-dockercfg-lfr7l\"" Oct 30 18:17:30.251582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.251559 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl"] Oct 30 18:17:30.372525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.367578 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7486973-71c0-4102-9f74-c814bf601ec5-istio-token\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.372525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.367687 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-istio-envoy\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.372525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.367753 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-workload-certs\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.372525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.367845 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-istio-data\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.372525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.367884 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwgj5\" (UniqueName: \"kubernetes.io/projected/f7486973-71c0-4102-9f74-c814bf601ec5-kube-api-access-qwgj5\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.372525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.367995 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-credential-socket\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.372525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.368060 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7486973-71c0-4102-9f74-c814bf601ec5-istio-podinfo\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.372525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.368150 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-workload-socket\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.372525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.368176 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7486973-71c0-4102-9f74-c814bf601ec5-istiod-ca-cert\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.410540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.407222 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:30.477575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.474878 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-istio-envoy\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.477575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.474963 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-workload-certs\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.477575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.475046 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-istio-data\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.477575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.475083 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwgj5\" (UniqueName: \"kubernetes.io/projected/f7486973-71c0-4102-9f74-c814bf601ec5-kube-api-access-qwgj5\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.477575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.475184 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-credential-socket\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.477575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.475246 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7486973-71c0-4102-9f74-c814bf601ec5-istio-podinfo\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.477575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.475329 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-workload-socket\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.477575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.475355 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7486973-71c0-4102-9f74-c814bf601ec5-istiod-ca-cert\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.477575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.475423 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7486973-71c0-4102-9f74-c814bf601ec5-istio-token\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.477575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.476688 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-credential-socket\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.477575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.476922 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-workload-certs\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.477575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.477150 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-istio-data\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.513543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.491313 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7486973-71c0-4102-9f74-c814bf601ec5-istiod-ca-cert\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.513543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.493676 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-workload-socket\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.531529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.529815 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-istio-envoy\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.531529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.530561 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7486973-71c0-4102-9f74-c814bf601ec5-istio-token\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.531529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.531038 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7486973-71c0-4102-9f74-c814bf601ec5-istio-podinfo\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.566168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.534263 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwgj5\" (UniqueName: \"kubernetes.io/projected/f7486973-71c0-4102-9f74-c814bf601ec5-kube-api-access-qwgj5\") pod \"e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.566168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.542271 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:30.585950 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.568518 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:30.904651 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:30.904611 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" event={"ID":"a6ae148a-f57c-4430-b5ee-c17a7885bc78","Type":"ContainerStarted","Data":"a3c3af46e76453edd4cf01ebfe87e5920ca5532087d932c13c4f6f9858f6ab4c"} Oct 30 18:17:31.067052 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:31.066742 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl"] Oct 30 18:17:31.101309 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:31.101273 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:31.126530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:31.101961 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:17:31.130735 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:31.130696 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:17:31.130986 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:31.130967 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:17:31.801325 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:31.801196 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:17:31.804719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:31.804687 4603 patch_prober.go:29] interesting pod/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:17:31.804984 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:31.804956 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" podUID="a6ae148a-f57c-4430-b5ee-c17a7885bc78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:17:31.917799 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:31.917758 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" event={"ID":"f7486973-71c0-4102-9f74-c814bf601ec5","Type":"ContainerStarted","Data":"c7b288f582bf7a4698353f293951d0b773c2be98ba32cb1f8abb2aa0de178bc3"} Oct 30 18:17:31.918264 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:31.917838 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" event={"ID":"f7486973-71c0-4102-9f74-c814bf601ec5","Type":"ContainerStarted","Data":"6b0b96201bec7c5b8b3215fc38161abed3e3bcdf7dcd779f80f98b411e987027"} Oct 30 18:17:31.967868 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:31.966900 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" podStartSLOduration=1.966874722 podStartE2EDuration="1.966874722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:31.962486709 +0000 UTC m=+1512.678056583" watchObservedRunningTime="2026-10-30 18:17:31.966874722 +0000 UTC m=+1512.682444597" Oct 30 18:17:32.543694 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:32.543649 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:32.549199 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:32.549173 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:32.746308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:32.746277 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:32.810713 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:32.810607 4603 patch_prober.go:29] interesting pod/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:17:32.810713 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:32.810674 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" podUID="a6ae148a-f57c-4430-b5ee-c17a7885bc78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:17:32.941803 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:32.941642 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:32.943085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:32.943065 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:17:32.956760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:32.956738 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:33.231358 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.231273 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:33.736794 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.736769 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp"] Oct 30 18:17:33.740049 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.740031 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:33.761256 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.761233 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-064408-openshift-default-dockercfg-48bvp\"" Oct 30 18:17:33.776774 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.776747 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp"] Oct 30 18:17:33.813528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.804853 4603 patch_prober.go:29] interesting pod/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:17:33.813528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.804908 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" podUID="a6ae148a-f57c-4430-b5ee-c17a7885bc78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:17:33.824188 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.824157 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-credential-socket\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:33.824312 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.824197 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-token\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:33.824312 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.824244 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chltx\" (UniqueName: \"kubernetes.io/projected/18da6086-657d-4917-9fcb-c8de74e7f7b4-kube-api-access-chltx\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:33.824442 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.824413 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18da6086-657d-4917-9fcb-c8de74e7f7b4-istiod-ca-cert\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:33.824479 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.824463 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-podinfo\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:33.824544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.824490 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-envoy\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:33.824702 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.824688 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-workload-certs\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:33.824742 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.824720 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-workload-socket\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:33.824800 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.824788 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-data\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:33.837694 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.837672 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:33.970531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.964775 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18da6086-657d-4917-9fcb-c8de74e7f7b4-istiod-ca-cert\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:33.970531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.964833 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-podinfo\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:33.970531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.964864 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-envoy\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:33.970531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.965036 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-workload-certs\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:33.970531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.965097 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-workload-socket\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:33.970531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.965227 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-data\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:33.970531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.965325 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-credential-socket\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:33.970531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.965351 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-token\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:33.970531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.965397 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chltx\" (UniqueName: \"kubernetes.io/projected/18da6086-657d-4917-9fcb-c8de74e7f7b4-kube-api-access-chltx\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:33.972344 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:33.972206 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-workload-certs\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:34.019578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:34.006133 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-podinfo\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:34.019578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:34.014001 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:34.021141 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:34.021108 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18da6086-657d-4917-9fcb-c8de74e7f7b4-istiod-ca-cert\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:34.021554 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:34.021536 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-data\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:34.021945 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:34.021926 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-credential-socket\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:34.036523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:34.022149 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-workload-socket\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:34.065526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:34.065397 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chltx\" (UniqueName: \"kubernetes.io/projected/18da6086-657d-4917-9fcb-c8de74e7f7b4-kube-api-access-chltx\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:34.078225 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:34.078158 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-token\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:34.078790 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:34.078766 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-envoy\") pod \"e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:34.357467 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:34.357385 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:34.614978 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:34.614884 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:34.725342 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:34.725320 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp"] Oct 30 18:17:34.727287 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:17:34.727262 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18da6086_657d_4917_9fcb_c8de74e7f7b4.slice/crio-67e8ca294440e0b44014a76749480935ecf333724f4746e22ec7d43cff798aaf WatchSource:0}: Error finding container 67e8ca294440e0b44014a76749480935ecf333724f4746e22ec7d43cff798aaf: Status 404 returned error can't find the container with id 67e8ca294440e0b44014a76749480935ecf333724f4746e22ec7d43cff798aaf Oct 30 18:17:34.729926 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:34.729893 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:17:34.730013 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:34.729961 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:17:34.730013 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:34.729990 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:17:34.805525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:34.803887 4603 patch_prober.go:29] interesting pod/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:17:34.805525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:34.803939 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" podUID="a6ae148a-f57c-4430-b5ee-c17a7885bc78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:17:34.824524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:34.817595 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:34.838289 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:34.834622 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:17:35.016896 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:35.012935 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:35.056076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:35.056013 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" event={"ID":"18da6086-657d-4917-9fcb-c8de74e7f7b4","Type":"ContainerStarted","Data":"3be733c96c1fbacee466c4189709eeeecf86becd478b651780424ad21f4e7a24"} Oct 30 18:17:35.056238 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:35.056123 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" event={"ID":"18da6086-657d-4917-9fcb-c8de74e7f7b4","Type":"ContainerStarted","Data":"67e8ca294440e0b44014a76749480935ecf333724f4746e22ec7d43cff798aaf"} Oct 30 18:17:35.370361 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:35.365912 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:35.370361 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:35.366425 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:17:35.370361 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:35.366649 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" podUID="18da6086-657d-4917-9fcb-c8de74e7f7b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 18:17:35.807528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:35.803869 4603 patch_prober.go:29] interesting pod/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:17:35.807528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:35.803923 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" podUID="a6ae148a-f57c-4430-b5ee-c17a7885bc78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:17:36.320970 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:36.320934 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:36.362687 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:36.362639 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:36.425733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:36.425681 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" podStartSLOduration=3.425665496 podStartE2EDuration="3.425665496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:35.117834835 +0000 UTC m=+1515.833404714" watchObservedRunningTime="2026-10-30 18:17:36.425665496 +0000 UTC m=+1517.141235369" Oct 30 18:17:36.480545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:36.480489 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:36.805529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:36.801856 4603 patch_prober.go:29] interesting pod/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:17:36.805529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:36.801912 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" podUID="a6ae148a-f57c-4430-b5ee-c17a7885bc78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:17:36.920628 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:36.920428 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:37.079269 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:37.079201 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:37.080609 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:37.080376 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:17:37.142345 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:37.142314 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:37.308122 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:37.308087 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:37.491998 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:37.491921 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2"] Oct 30 18:17:37.492398 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:37.492210 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" podUID="3bb952b3-3096-442b-8ab6-2909cee2c021" containerName="istio-proxy" containerID="cri-o://cd0c93c47c474940e8505e75028c6d6d4340adfa764f47dcd83af08dde324fda" gracePeriod=30 Oct 30 18:17:37.542304 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:37.542273 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:37.695142 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:37.695107 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:37.809210 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:37.809177 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:17:37.812998 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:37.812973 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:17:37.865121 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:37.865091 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:38.718208 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:38.718175 4603 patch_prober.go:29] interesting pod/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:17:38.718649 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:38.718240 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" podUID="3bb952b3-3096-442b-8ab6-2909cee2c021" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:17:38.937409 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:38.937371 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:39.459285 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:39.459156 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:39.625583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:39.625546 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:40.867719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:40.867686 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:41.026602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:41.026569 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:41.698487 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:41.698447 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:41.849032 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:41.848998 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:42.706963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.706922 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:42.887525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.879489 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:17:42.887525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.880254 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:42.994328 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.994298 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-data\") pod \"3bb952b3-3096-442b-8ab6-2909cee2c021\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " Oct 30 18:17:42.994534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.994441 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-workload-socket\") pod \"3bb952b3-3096-442b-8ab6-2909cee2c021\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " Oct 30 18:17:42.994534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.994471 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-podinfo\") pod \"3bb952b3-3096-442b-8ab6-2909cee2c021\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " Oct 30 18:17:42.994676 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.994613 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-data" pod "3bb952b3-3096-442b-8ab6-2909cee2c021" (UID: "3bb952b3-3096-442b-8ab6-2909cee2c021"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:17:42.994739 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.994702 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-token\" (UniqueName: \"kubernetes.io/projected/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-token\") pod \"3bb952b3-3096-442b-8ab6-2909cee2c021\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " Oct 30 18:17:42.994739 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.994732 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bb952b3-3096-442b-8ab6-2909cee2c021-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb952b3-3096-442b-8ab6-2909cee2c021-istiod-ca-cert\") pod \"3bb952b3-3096-442b-8ab6-2909cee2c021\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " Oct 30 18:17:42.994833 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.994752 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-workload-socket" pod "3bb952b3-3096-442b-8ab6-2909cee2c021" (UID: "3bb952b3-3096-442b-8ab6-2909cee2c021"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:17:42.994833 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.994764 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-envoy\") pod \"3bb952b3-3096-442b-8ab6-2909cee2c021\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " Oct 30 18:17:42.994833 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.994801 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-credential-socket\") pod \"3bb952b3-3096-442b-8ab6-2909cee2c021\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " Oct 30 18:17:42.994998 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.994837 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-workload-certs\") pod \"3bb952b3-3096-442b-8ab6-2909cee2c021\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " Oct 30 18:17:42.994998 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.994882 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb952b3-3096-442b-8ab6-2909cee2c021-kube-api-access-ctdkv\" (UniqueName: \"kubernetes.io/projected/3bb952b3-3096-442b-8ab6-2909cee2c021-kube-api-access-ctdkv\") pod \"3bb952b3-3096-442b-8ab6-2909cee2c021\" (UID: \"3bb952b3-3096-442b-8ab6-2909cee2c021\") " Oct 30 18:17:42.995109 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.995052 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-credential-socket" pod "3bb952b3-3096-442b-8ab6-2909cee2c021" (UID: "3bb952b3-3096-442b-8ab6-2909cee2c021"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:17:42.995162 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.995138 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-workload-certs" pod "3bb952b3-3096-442b-8ab6-2909cee2c021" (UID: "3bb952b3-3096-442b-8ab6-2909cee2c021"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:17:42.995206 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.995176 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb952b3-3096-442b-8ab6-2909cee2c021-istiod-ca-cert" pod "3bb952b3-3096-442b-8ab6-2909cee2c021" (UID: "3bb952b3-3096-442b-8ab6-2909cee2c021"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:42.995814 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.995792 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:17:42.995867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.995826 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb952b3-3096-442b-8ab6-2909cee2c021-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:17:42.995867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.995844 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:17:42.995867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.995859 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:17:42.995966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.995875 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:17:42.997032 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.997007 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-podinfo" pod "3bb952b3-3096-442b-8ab6-2909cee2c021" (UID: "3bb952b3-3096-442b-8ab6-2909cee2c021"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:17:42.997176 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.997053 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb952b3-3096-442b-8ab6-2909cee2c021-kube-api-access-ctdkv" pod "3bb952b3-3096-442b-8ab6-2909cee2c021" (UID: "3bb952b3-3096-442b-8ab6-2909cee2c021"). InnerVolumeSpecName "kube-api-access-ctdkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:42.997448 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.997427 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-token" pod "3bb952b3-3096-442b-8ab6-2909cee2c021" (UID: "3bb952b3-3096-442b-8ab6-2909cee2c021"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:42.997878 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:42.997863 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-envoy" pod "3bb952b3-3096-442b-8ab6-2909cee2c021" (UID: "3bb952b3-3096-442b-8ab6-2909cee2c021"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:17:43.024169 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:43.024142 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:43.097264 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:43.097240 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:17:43.097264 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:43.097262 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:17:43.097400 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:43.097272 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb952b3-3096-442b-8ab6-2909cee2c021-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:17:43.097400 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:43.097280 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctdkv\" (UniqueName: \"kubernetes.io/projected/3bb952b3-3096-442b-8ab6-2909cee2c021-kube-api-access-ctdkv\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:17:43.204217 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:43.204148 4603 generic.go:356] "Generic (PLEG): container finished" podID="3bb952b3-3096-442b-8ab6-2909cee2c021" containerID="cd0c93c47c474940e8505e75028c6d6d4340adfa764f47dcd83af08dde324fda" exitCode=0 Oct 30 18:17:43.204335 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:43.204222 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" Oct 30 18:17:43.204335 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:43.204236 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" event={"ID":"3bb952b3-3096-442b-8ab6-2909cee2c021","Type":"ContainerDied","Data":"cd0c93c47c474940e8505e75028c6d6d4340adfa764f47dcd83af08dde324fda"} Oct 30 18:17:43.204335 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:43.204303 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2" event={"ID":"3bb952b3-3096-442b-8ab6-2909cee2c021","Type":"ContainerDied","Data":"16e533922b4625bb49b92b4523ba2bc9e0c02b2da4a1110e19a0cec366eea5eb"} Oct 30 18:17:43.204473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:43.204338 4603 scope.go:122] "RemoveContainer" containerID="cd0c93c47c474940e8505e75028c6d6d4340adfa764f47dcd83af08dde324fda" Oct 30 18:17:43.218725 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:43.218708 4603 scope.go:122] "RemoveContainer" containerID="29f2db7a9fed86224bd7bbf5685d606ede4d5b3a0c5a6fff0f036271f11bff75" Oct 30 18:17:43.228210 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:43.228191 4603 scope.go:122] "RemoveContainer" containerID="cd0c93c47c474940e8505e75028c6d6d4340adfa764f47dcd83af08dde324fda" Oct 30 18:17:43.228493 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:17:43.228475 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd0c93c47c474940e8505e75028c6d6d4340adfa764f47dcd83af08dde324fda\": container with ID starting with cd0c93c47c474940e8505e75028c6d6d4340adfa764f47dcd83af08dde324fda not found: ID does not exist" containerID="cd0c93c47c474940e8505e75028c6d6d4340adfa764f47dcd83af08dde324fda" Oct 30 18:17:43.228585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:43.228531 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd0c93c47c474940e8505e75028c6d6d4340adfa764f47dcd83af08dde324fda"} err="failed to get container status \"cd0c93c47c474940e8505e75028c6d6d4340adfa764f47dcd83af08dde324fda\": rpc error: code = NotFound desc = could not find container \"cd0c93c47c474940e8505e75028c6d6d4340adfa764f47dcd83af08dde324fda\": container with ID starting with cd0c93c47c474940e8505e75028c6d6d4340adfa764f47dcd83af08dde324fda not found: ID does not exist" Oct 30 18:17:43.228585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:43.228551 4603 scope.go:122] "RemoveContainer" containerID="29f2db7a9fed86224bd7bbf5685d606ede4d5b3a0c5a6fff0f036271f11bff75" Oct 30 18:17:43.228820 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:17:43.228804 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f2db7a9fed86224bd7bbf5685d606ede4d5b3a0c5a6fff0f036271f11bff75\": container with ID starting with 29f2db7a9fed86224bd7bbf5685d606ede4d5b3a0c5a6fff0f036271f11bff75 not found: ID does not exist" containerID="29f2db7a9fed86224bd7bbf5685d606ede4d5b3a0c5a6fff0f036271f11bff75" Oct 30 18:17:43.228869 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:43.228822 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f2db7a9fed86224bd7bbf5685d606ede4d5b3a0c5a6fff0f036271f11bff75"} err="failed to get container status \"29f2db7a9fed86224bd7bbf5685d606ede4d5b3a0c5a6fff0f036271f11bff75\": rpc error: code = NotFound desc = could not find container \"29f2db7a9fed86224bd7bbf5685d606ede4d5b3a0c5a6fff0f036271f11bff75\": container with ID starting with 29f2db7a9fed86224bd7bbf5685d606ede4d5b3a0c5a6fff0f036271f11bff75 not found: ID does not exist" Oct 30 18:17:43.256487 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:43.256462 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2"] Oct 30 18:17:43.268114 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:43.268092 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c8f010-openshift-default-5dd944d9b9-4m4p2"] Oct 30 18:17:43.410143 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:43.410115 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bb952b3-3096-442b-8ab6-2909cee2c021" path="/var/lib/kubelet/pods/3bb952b3-3096-442b-8ab6-2909cee2c021/volumes" Oct 30 18:17:44.584958 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:44.584921 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:46.235325 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:46.235282 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:46.994993 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:46.994950 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:47.427247 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.427208 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl"] Oct 30 18:17:47.428088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.428069 4603 cpu_manager.go:414] "containerMap: removing container" podUID="3bb952b3-3096-442b-8ab6-2909cee2c021" containerName="istio-proxy" Oct 30 18:17:47.428193 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.428089 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb952b3-3096-442b-8ab6-2909cee2c021" containerName="istio-proxy" Oct 30 18:17:47.428193 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.428112 4603 cpu_manager.go:414] "containerMap: removing container" podUID="3bb952b3-3096-442b-8ab6-2909cee2c021" containerName="istio-proxy" Oct 30 18:17:47.428193 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.428121 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb952b3-3096-442b-8ab6-2909cee2c021" containerName="istio-proxy" Oct 30 18:17:47.428349 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.428287 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb952b3-3096-442b-8ab6-2909cee2c021" containerName="istio-proxy" Oct 30 18:17:47.433933 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.433907 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.454792 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.454768 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl"] Oct 30 18:17:47.462566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.462486 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c33ee4-openshift-default-dockercfg-vvlfm\"" Oct 30 18:17:47.480842 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.480814 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d036618a-de13-4ab8-affb-7fc490084530-istio-token\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.481009 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.480864 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-istio-envoy\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.481320 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.481294 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d036618a-de13-4ab8-affb-7fc490084530-istiod-ca-cert\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.481423 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.481355 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-workload-certs\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.481628 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.481479 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-workload-socket\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.481628 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.481582 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d036618a-de13-4ab8-affb-7fc490084530-istio-podinfo\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.481902 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.481877 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5rzq\" (UniqueName: \"kubernetes.io/projected/d036618a-de13-4ab8-affb-7fc490084530-kube-api-access-x5rzq\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.481975 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.481943 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-credential-socket\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.482034 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.481989 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-istio-data\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.532214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.532193 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:47.583657 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.583633 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-workload-socket\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.583776 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.583696 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d036618a-de13-4ab8-affb-7fc490084530-istio-podinfo\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.583829 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.583812 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5rzq\" (UniqueName: \"kubernetes.io/projected/d036618a-de13-4ab8-affb-7fc490084530-kube-api-access-x5rzq\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.583873 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.583849 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-credential-socket\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.583927 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.583897 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-istio-data\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.584021 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.584001 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d036618a-de13-4ab8-affb-7fc490084530-istio-token\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.584108 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.584047 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-istio-envoy\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.584172 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.584106 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-workload-socket\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.584172 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.584126 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d036618a-de13-4ab8-affb-7fc490084530-istiod-ca-cert\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.584172 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.584155 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-workload-certs\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.584338 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.584269 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-istio-data\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.584554 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.584535 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-workload-certs\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.584633 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.584595 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-credential-socket\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.584864 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.584845 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d036618a-de13-4ab8-affb-7fc490084530-istiod-ca-cert\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.586425 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.586406 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-istio-envoy\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.586876 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.586855 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d036618a-de13-4ab8-affb-7fc490084530-istio-podinfo\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.616229 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.616101 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d036618a-de13-4ab8-affb-7fc490084530-istio-token\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.617779 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.617706 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5rzq\" (UniqueName: \"kubernetes.io/projected/d036618a-de13-4ab8-affb-7fc490084530-kube-api-access-x5rzq\") pod \"e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:47.685557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.685363 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:47.752940 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:47.752914 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:48.112343 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:48.112311 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:48.249051 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:48.249021 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl"] Oct 30 18:17:48.249353 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:17:48.249329 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd036618a_de13_4ab8_affb_7fc490084530.slice/crio-2c6c5ca6f27fa765ab44370c8dc6b6effe3607df05fe81f6c87ec0de333143f1 WatchSource:0}: Error finding container 2c6c5ca6f27fa765ab44370c8dc6b6effe3607df05fe81f6c87ec0de333143f1: Status 404 returned error can't find the container with id 2c6c5ca6f27fa765ab44370c8dc6b6effe3607df05fe81f6c87ec0de333143f1 Oct 30 18:17:48.251885 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:48.251853 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:17:48.251994 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:48.251926 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:17:48.251994 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:48.251971 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:17:48.800323 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:48.800295 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:48.949870 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:48.948045 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:49.255234 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:49.255159 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" event={"ID":"d036618a-de13-4ab8-affb-7fc490084530","Type":"ContainerStarted","Data":"9936aa0093b73d9f0acdb8ae8c2a7ee8b9f9bf2b7b35147dcc208186c9b3bb01"} Oct 30 18:17:49.255234 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:49.255216 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" event={"ID":"d036618a-de13-4ab8-affb-7fc490084530","Type":"ContainerStarted","Data":"2c6c5ca6f27fa765ab44370c8dc6b6effe3607df05fe81f6c87ec0de333143f1"} Oct 30 18:17:49.819165 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:49.818962 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:49.826149 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:49.825593 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:49.827280 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:49.827134 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:49.878760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:49.878464 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" podStartSLOduration=2.87844439 podStartE2EDuration="2.87844439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:49.360310197 +0000 UTC m=+1530.075880075" watchObservedRunningTime="2026-10-30 18:17:49.87844439 +0000 UTC m=+1530.594014267" Oct 30 18:17:49.977020 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:49.976978 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:50.121912 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:50.121575 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:50.261167 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:50.261136 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:50.262265 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:50.262244 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:17:51.346099 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:51.346071 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:51.496616 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:51.496586 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:51.640837 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:51.640753 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:51.930893 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:17:51.930811 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:17:53.193704 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:53.193669 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:53.344526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:53.338400 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:53.485751 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:53.485674 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:54.948214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:54.948179 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:56.711266 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:56.711235 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:58.336115 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:58.336078 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:59.990996 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:17:59.990958 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:01.764363 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:01.764320 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:02.114022 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:02.113992 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:02.251708 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:02.251674 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:03.739423 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:03.739388 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:03.882211 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:03.882173 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:03.976939 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:03.976911 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw"] Oct 30 18:18:03.977613 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:03.977598 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb952b3-3096-442b-8ab6-2909cee2c021" containerName="istio-proxy" Oct 30 18:18:03.981169 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:03.981150 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw" Oct 30 18:18:03.987121 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:03.987097 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7325-7cdd3a"] Oct 30 18:18:03.991089 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:03.990847 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-7cdd3a" Oct 30 18:18:03.997731 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:03.997697 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c33ee4\"" Oct 30 18:18:04.011894 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.011871 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c33ee4-dockercfg-m9vmn\"" Oct 30 18:18:04.015673 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.015656 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw"] Oct 30 18:18:04.018659 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.018641 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7325-7cdd3a"] Oct 30 18:18:04.030805 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.030780 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:04.114525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.114482 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 18:18:04.114853 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.114836 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vdx7j\"" Oct 30 18:18:04.115116 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.115103 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 18:18:04.268587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.254936 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cb7f\" (UniqueName: \"kubernetes.io/projected/ecac0277-6449-4646-ab2c-918696deddf0-kube-api-access-6cb7f\") pod \"payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw\" (UID: \"ecac0277-6449-4646-ab2c-918696deddf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw" Oct 30 18:18:04.268587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.255041 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qrxd\" (UniqueName: \"kubernetes.io/projected/d06b6b3c-e138-4478-b7da-48e4769068ae-kube-api-access-7qrxd\") pod \"test-curl-7325-7cdd3a\" (UID: \"d06b6b3c-e138-4478-b7da-48e4769068ae\") " pod="opendatahub/test-curl-7325-7cdd3a" Oct 30 18:18:04.268587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.255185 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecac0277-6449-4646-ab2c-918696deddf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw\" (UID: \"ecac0277-6449-4646-ab2c-918696deddf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw" Oct 30 18:18:04.361995 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.361956 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cb7f\" (UniqueName: \"kubernetes.io/projected/ecac0277-6449-4646-ab2c-918696deddf0-kube-api-access-6cb7f\") pod \"payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw\" (UID: \"ecac0277-6449-4646-ab2c-918696deddf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw" Oct 30 18:18:04.362548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.362496 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qrxd\" (UniqueName: \"kubernetes.io/projected/d06b6b3c-e138-4478-b7da-48e4769068ae-kube-api-access-7qrxd\") pod \"test-curl-7325-7cdd3a\" (UID: \"d06b6b3c-e138-4478-b7da-48e4769068ae\") " pod="opendatahub/test-curl-7325-7cdd3a" Oct 30 18:18:04.376950 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.367849 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecac0277-6449-4646-ab2c-918696deddf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw\" (UID: \"ecac0277-6449-4646-ab2c-918696deddf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw" Oct 30 18:18:04.377272 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.371282 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecac0277-6449-4646-ab2c-918696deddf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw\" (UID: \"ecac0277-6449-4646-ab2c-918696deddf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw" Oct 30 18:18:04.381803 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.379577 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69"] Oct 30 18:18:04.393019 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.385659 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69" Oct 30 18:18:04.393019 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.390587 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69"] Oct 30 18:18:04.413682 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.413650 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cb7f\" (UniqueName: \"kubernetes.io/projected/ecac0277-6449-4646-ab2c-918696deddf0-kube-api-access-6cb7f\") pod \"payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw\" (UID: \"ecac0277-6449-4646-ab2c-918696deddf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw" Oct 30 18:18:04.414164 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.414143 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qrxd\" (UniqueName: \"kubernetes.io/projected/d06b6b3c-e138-4478-b7da-48e4769068ae-kube-api-access-7qrxd\") pod \"test-curl-7325-7cdd3a\" (UID: \"d06b6b3c-e138-4478-b7da-48e4769068ae\") " pod="opendatahub/test-curl-7325-7cdd3a" Oct 30 18:18:04.479048 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.479014 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qmfb\" (UniqueName: \"kubernetes.io/projected/050e2617-3912-482f-a910-c918b14efb6c-kube-api-access-9qmfb\") pod \"payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69\" (UID: \"050e2617-3912-482f-a910-c918b14efb6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69" Oct 30 18:18:04.479193 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.479174 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050e2617-3912-482f-a910-c918b14efb6c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69\" (UID: \"050e2617-3912-482f-a910-c918b14efb6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69" Oct 30 18:18:04.580926 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.580899 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qmfb\" (UniqueName: \"kubernetes.io/projected/050e2617-3912-482f-a910-c918b14efb6c-kube-api-access-9qmfb\") pod \"payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69\" (UID: \"050e2617-3912-482f-a910-c918b14efb6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69" Oct 30 18:18:04.581065 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.580999 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050e2617-3912-482f-a910-c918b14efb6c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69\" (UID: \"050e2617-3912-482f-a910-c918b14efb6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69" Oct 30 18:18:04.581733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.581711 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050e2617-3912-482f-a910-c918b14efb6c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69\" (UID: \"050e2617-3912-482f-a910-c918b14efb6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69" Oct 30 18:18:04.601729 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.601712 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw" Oct 30 18:18:04.604168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.604148 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qmfb\" (UniqueName: \"kubernetes.io/projected/050e2617-3912-482f-a910-c918b14efb6c-kube-api-access-9qmfb\") pod \"payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69\" (UID: \"050e2617-3912-482f-a910-c918b14efb6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69" Oct 30 18:18:04.616263 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.616241 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-7cdd3a" Oct 30 18:18:04.725939 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.725900 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69" Oct 30 18:18:04.907819 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.907796 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw"] Oct 30 18:18:04.908518 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:18:04.908481 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecac0277_6449_4646_ab2c_918696deddf0.slice/crio-8bc07cacaad7b135f36a21255f8abcd1c3219759b26d472c5a2fa4846e233664 WatchSource:0}: Error finding container 8bc07cacaad7b135f36a21255f8abcd1c3219759b26d472c5a2fa4846e233664: Status 404 returned error can't find the container with id 8bc07cacaad7b135f36a21255f8abcd1c3219759b26d472c5a2fa4846e233664 Oct 30 18:18:04.923667 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:04.923638 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7325-7cdd3a"] Oct 30 18:18:04.923848 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:18:04.923822 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd06b6b3c_e138_4478_b7da_48e4769068ae.slice/crio-cdf23ada92e7628298898642d4e2ef10361d71dd3ef95024a00bd4c44dc8bdff WatchSource:0}: Error finding container cdf23ada92e7628298898642d4e2ef10361d71dd3ef95024a00bd4c44dc8bdff: Status 404 returned error can't find the container with id cdf23ada92e7628298898642d4e2ef10361d71dd3ef95024a00bd4c44dc8bdff Oct 30 18:18:05.038041 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:05.038014 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69"] Oct 30 18:18:05.039317 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:18:05.039292 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050e2617_3912_482f_a910_c918b14efb6c.slice/crio-f24bf05e8b8436d028a25c046316a87a34250b8dbcbf833dfa744260323d1ad3 WatchSource:0}: Error finding container f24bf05e8b8436d028a25c046316a87a34250b8dbcbf833dfa744260323d1ad3: Status 404 returned error can't find the container with id f24bf05e8b8436d028a25c046316a87a34250b8dbcbf833dfa744260323d1ad3 Oct 30 18:18:05.464419 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:05.464385 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7325-7cdd3a" event={"ID":"d06b6b3c-e138-4478-b7da-48e4769068ae","Type":"ContainerStarted","Data":"cc2b4a236a9106ff5efdffeb41aef849ecf45d663d434190b95a19188bdb381a"} Oct 30 18:18:05.464618 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:05.464442 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7325-7cdd3a" event={"ID":"d06b6b3c-e138-4478-b7da-48e4769068ae","Type":"ContainerStarted","Data":"cdf23ada92e7628298898642d4e2ef10361d71dd3ef95024a00bd4c44dc8bdff"} Oct 30 18:18:05.466042 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:05.466015 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69" event={"ID":"050e2617-3912-482f-a910-c918b14efb6c","Type":"ContainerStarted","Data":"fc4a5fb2ac7bff99a1e94ab22a28c400434d0d7f6fbf7398477a2e917a70911b"} Oct 30 18:18:05.466152 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:05.466074 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69" event={"ID":"050e2617-3912-482f-a910-c918b14efb6c","Type":"ContainerStarted","Data":"f24bf05e8b8436d028a25c046316a87a34250b8dbcbf833dfa744260323d1ad3"} Oct 30 18:18:05.466152 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:05.466106 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69" Oct 30 18:18:05.467603 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:05.467581 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw" event={"ID":"ecac0277-6449-4646-ab2c-918696deddf0","Type":"ContainerStarted","Data":"38b615730eaa08526bb7d05d2eb0c2028416d9016c4caf469c42ad8942da9eef"} Oct 30 18:18:05.467692 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:05.467618 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw" event={"ID":"ecac0277-6449-4646-ab2c-918696deddf0","Type":"ContainerStarted","Data":"8bc07cacaad7b135f36a21255f8abcd1c3219759b26d472c5a2fa4846e233664"} Oct 30 18:18:05.467730 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:05.467697 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw" Oct 30 18:18:05.503981 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:05.503939 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7325-7cdd3a" podStartSLOduration=2.503926233 podStartE2EDuration="2.503926233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:05.501422722 +0000 UTC m=+1546.216992595" watchObservedRunningTime="2026-10-30 18:18:05.503926233 +0000 UTC m=+1546.219496107" Oct 30 18:18:05.568151 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:05.545167 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw" podStartSLOduration=2.545147482 podStartE2EDuration="2.545147482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:05.542043204 +0000 UTC m=+1546.257613078" watchObservedRunningTime="2026-10-30 18:18:05.545147482 +0000 UTC m=+1546.260717360" Oct 30 18:18:05.576256 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:05.569795 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:06.439277 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:06.439219 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69" podStartSLOduration=2.4392009789999998 podStartE2EDuration="2.439200979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:05.609789175 +0000 UTC m=+1546.325359051" watchObservedRunningTime="2026-10-30 18:18:06.439200979 +0000 UTC m=+1547.154770854" Oct 30 18:18:06.441217 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:06.441190 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7325-7cdd3a"] Oct 30 18:18:06.445833 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:06.445813 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7325-7cdd3a"] Oct 30 18:18:07.354452 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:07.354411 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:07.500992 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:07.500952 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7325-7cdd3a" podUID="d06b6b3c-e138-4478-b7da-48e4769068ae" containerName="test-curl-7325-7cdd3a" containerID="cri-o://cc2b4a236a9106ff5efdffeb41aef849ecf45d663d434190b95a19188bdb381a" gracePeriod=2 Oct 30 18:18:09.292866 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:09.292829 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:09.719941 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:09.719918 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-7cdd3a" Oct 30 18:18:09.801182 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:09.801121 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d06b6b3c-e138-4478-b7da-48e4769068ae-kube-api-access-7qrxd\" (UniqueName: \"kubernetes.io/projected/d06b6b3c-e138-4478-b7da-48e4769068ae-kube-api-access-7qrxd\") pod \"d06b6b3c-e138-4478-b7da-48e4769068ae\" (UID: \"d06b6b3c-e138-4478-b7da-48e4769068ae\") " Oct 30 18:18:09.803271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:09.803247 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d06b6b3c-e138-4478-b7da-48e4769068ae-kube-api-access-7qrxd" pod "d06b6b3c-e138-4478-b7da-48e4769068ae" (UID: "d06b6b3c-e138-4478-b7da-48e4769068ae"). InnerVolumeSpecName "kube-api-access-7qrxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:09.903168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:09.903145 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qrxd\" (UniqueName: \"kubernetes.io/projected/d06b6b3c-e138-4478-b7da-48e4769068ae-kube-api-access-7qrxd\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:18:10.277566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:10.277534 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:10.447531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:10.442636 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:10.551169 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:10.551076 4603 generic.go:356] "Generic (PLEG): container finished" podID="d06b6b3c-e138-4478-b7da-48e4769068ae" containerID="cc2b4a236a9106ff5efdffeb41aef849ecf45d663d434190b95a19188bdb381a" exitCode=137 Oct 30 18:18:10.551475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:10.551452 4603 scope.go:122] "RemoveContainer" containerID="cc2b4a236a9106ff5efdffeb41aef849ecf45d663d434190b95a19188bdb381a" Oct 30 18:18:10.551841 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:10.551821 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-7cdd3a" Oct 30 18:18:10.612099 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:10.612004 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:10.637489 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:10.637467 4603 scope.go:122] "RemoveContainer" containerID="cc2b4a236a9106ff5efdffeb41aef849ecf45d663d434190b95a19188bdb381a" Oct 30 18:18:10.637865 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:18:10.637843 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc2b4a236a9106ff5efdffeb41aef849ecf45d663d434190b95a19188bdb381a\": container with ID starting with cc2b4a236a9106ff5efdffeb41aef849ecf45d663d434190b95a19188bdb381a not found: ID does not exist" containerID="cc2b4a236a9106ff5efdffeb41aef849ecf45d663d434190b95a19188bdb381a" Oct 30 18:18:10.637955 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:10.637876 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc2b4a236a9106ff5efdffeb41aef849ecf45d663d434190b95a19188bdb381a"} err="failed to get container status \"cc2b4a236a9106ff5efdffeb41aef849ecf45d663d434190b95a19188bdb381a\": rpc error: code = NotFound desc = could not find container \"cc2b4a236a9106ff5efdffeb41aef849ecf45d663d434190b95a19188bdb381a\": container with ID starting with cc2b4a236a9106ff5efdffeb41aef849ecf45d663d434190b95a19188bdb381a not found: ID does not exist" Oct 30 18:18:10.644947 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:18:10.644922 4603 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7325-7cdd3a\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="d06b6b3c-e138-4478-b7da-48e4769068ae" pod="opendatahub/test-curl-7325-7cdd3a" Oct 30 18:18:10.958584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:10.954148 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:11.409995 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:11.409964 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d06b6b3c-e138-4478-b7da-48e4769068ae" path="/var/lib/kubelet/pods/d06b6b3c-e138-4478-b7da-48e4769068ae/volumes" Oct 30 18:18:12.315308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:12.315266 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf"] Oct 30 18:18:12.315833 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:12.315601 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" podUID="5a6d0fd2-c58a-4df1-95c2-754554fd69d3" containerName="maas-api" containerID="cri-o://0a16a67b1b52599e53682472f8c9c078fb7346501d5eeecd2e06d394eba9ebcb" gracePeriod=30 Oct 30 18:18:12.411781 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:12.411746 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:12.813753 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:12.813717 4603 generic.go:356] "Generic (PLEG): container finished" podID="5a6d0fd2-c58a-4df1-95c2-754554fd69d3" containerID="0a16a67b1b52599e53682472f8c9c078fb7346501d5eeecd2e06d394eba9ebcb" exitCode=0 Oct 30 18:18:12.814103 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:12.814079 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" event={"ID":"5a6d0fd2-c58a-4df1-95c2-754554fd69d3","Type":"ContainerDied","Data":"0a16a67b1b52599e53682472f8c9c078fb7346501d5eeecd2e06d394eba9ebcb"} Oct 30 18:18:12.941525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:12.929296 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:12.945524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:12.944900 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" Oct 30 18:18:12.952521 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:12.948890 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-kube-api-access-ltz7r\" (UniqueName: \"kubernetes.io/projected/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-kube-api-access-ltz7r\") pod \"5a6d0fd2-c58a-4df1-95c2-754554fd69d3\" (UID: \"5a6d0fd2-c58a-4df1-95c2-754554fd69d3\") " Oct 30 18:18:12.952521 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:12.948956 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-maas-api-tls\") pod \"5a6d0fd2-c58a-4df1-95c2-754554fd69d3\" (UID: \"5a6d0fd2-c58a-4df1-95c2-754554fd69d3\") " Oct 30 18:18:12.952521 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:12.949062 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-openshift-service-ca\") pod \"5a6d0fd2-c58a-4df1-95c2-754554fd69d3\" (UID: \"5a6d0fd2-c58a-4df1-95c2-754554fd69d3\") " Oct 30 18:18:12.952521 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:12.949128 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-metrics-certs\") pod \"5a6d0fd2-c58a-4df1-95c2-754554fd69d3\" (UID: \"5a6d0fd2-c58a-4df1-95c2-754554fd69d3\") " Oct 30 18:18:12.952521 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:12.950862 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-openshift-service-ca" pod "5a6d0fd2-c58a-4df1-95c2-754554fd69d3" (UID: "5a6d0fd2-c58a-4df1-95c2-754554fd69d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:12.956959 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:12.955789 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-kube-api-access-ltz7r" pod "5a6d0fd2-c58a-4df1-95c2-754554fd69d3" (UID: "5a6d0fd2-c58a-4df1-95c2-754554fd69d3"). InnerVolumeSpecName "kube-api-access-ltz7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:12.980880 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:12.957357 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-metrics-certs" pod "5a6d0fd2-c58a-4df1-95c2-754554fd69d3" (UID: "5a6d0fd2-c58a-4df1-95c2-754554fd69d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:12.994835 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:12.993670 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-maas-api-tls" pod "5a6d0fd2-c58a-4df1-95c2-754554fd69d3" (UID: "5a6d0fd2-c58a-4df1-95c2-754554fd69d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:13.074604 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:13.055764 4603 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-openshift-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:18:13.074604 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:13.055797 4603 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:18:13.074604 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:13.055814 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltz7r\" (UniqueName: \"kubernetes.io/projected/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-kube-api-access-ltz7r\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:18:13.074604 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:13.055828 4603 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a6d0fd2-c58a-4df1-95c2-754554fd69d3-maas-api-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:18:13.102535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:13.080661 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:13.355783 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:13.355690 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:13.517927 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:13.517897 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:13.651978 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:13.651901 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:13.822749 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:13.822722 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" Oct 30 18:18:13.822927 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:13.822750 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf" event={"ID":"5a6d0fd2-c58a-4df1-95c2-754554fd69d3","Type":"ContainerDied","Data":"cdfd74d793b084b563a08861b6d6dd54dc7b76701ac3ffb40a9b31537d254557"} Oct 30 18:18:13.822927 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:13.822803 4603 scope.go:122] "RemoveContainer" containerID="0a16a67b1b52599e53682472f8c9c078fb7346501d5eeecd2e06d394eba9ebcb" Oct 30 18:18:13.875043 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:13.875021 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf"] Oct 30 18:18:13.886575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:13.886552 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da151d-6b88bc8445-q92zf"] Oct 30 18:18:15.043432 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:15.043402 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:15.191846 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:15.191810 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:15.410190 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:15.410103 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a6d0fd2-c58a-4df1-95c2-754554fd69d3" path="/var/lib/kubelet/pods/5a6d0fd2-c58a-4df1-95c2-754554fd69d3/volumes" Oct 30 18:18:16.494778 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:16.494750 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw" Oct 30 18:18:16.495163 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:16.494843 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69" Oct 30 18:18:16.639468 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:16.639440 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:17.463678 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:17.463643 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:17.633195 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:17.632618 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:18.417609 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:18.417572 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:18.708213 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:18.708120 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:20.093680 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:20.093644 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:20.247001 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:20.246909 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:20.399845 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:20.399766 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr"] Oct 30 18:18:20.400328 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:20.400277 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" podUID="a6ae148a-f57c-4430-b5ee-c17a7885bc78" containerName="istio-proxy" containerID="cri-o://a3c3af46e76453edd4cf01ebfe87e5920ca5532087d932c13c4f6f9858f6ab4c" gracePeriod=30 Oct 30 18:18:20.450311 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:20.450285 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:21.474566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:21.474535 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:21.851219 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:21.851191 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww"] Oct 30 18:18:21.851809 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:21.851789 4603 cpu_manager.go:414] "containerMap: removing container" podUID="5a6d0fd2-c58a-4df1-95c2-754554fd69d3" containerName="maas-api" Oct 30 18:18:21.851809 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:21.851805 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a6d0fd2-c58a-4df1-95c2-754554fd69d3" containerName="maas-api" Oct 30 18:18:21.851998 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:21.851836 4603 cpu_manager.go:414] "containerMap: removing container" podUID="d06b6b3c-e138-4478-b7da-48e4769068ae" containerName="test-curl-7325-7cdd3a" Oct 30 18:18:21.851998 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:21.851845 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d06b6b3c-e138-4478-b7da-48e4769068ae" containerName="test-curl-7325-7cdd3a" Oct 30 18:18:21.852112 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:21.852015 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a6d0fd2-c58a-4df1-95c2-754554fd69d3" containerName="maas-api" Oct 30 18:18:21.852112 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:21.852032 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="d06b6b3c-e138-4478-b7da-48e4769068ae" containerName="test-curl-7325-7cdd3a" Oct 30 18:18:21.857997 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:21.857961 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:21.873475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:21.873447 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d951c4-openshift-default-dockercfg-tgrdn\"" Oct 30 18:18:21.890992 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:21.890951 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww"] Oct 30 18:18:21.905825 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:21.905758 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-credential-socket\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:21.906039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:21.906017 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-token\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:21.906234 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:21.906214 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6vwk\" (UniqueName: \"kubernetes.io/projected/30b23750-56cb-4895-80a8-8e54ae819f7a-kube-api-access-n6vwk\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:21.906401 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:21.906383 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-workload-certs\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:21.906624 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:21.906605 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-envoy\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:21.906793 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:21.906775 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b23750-56cb-4895-80a8-8e54ae819f7a-istiod-ca-cert\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:21.906915 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:21.906898 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-workload-socket\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:21.907066 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:21.907048 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-data\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:21.907432 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:21.907409 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-podinfo\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:21.939433 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:18:21.939400 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:18:21.941812 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:21.941786 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:22.054995 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.054955 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-credential-socket\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:22.055241 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.055223 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-token\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:22.055413 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.055398 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6vwk\" (UniqueName: \"kubernetes.io/projected/30b23750-56cb-4895-80a8-8e54ae819f7a-kube-api-access-n6vwk\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:22.055612 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.055581 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-workload-certs\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:22.087677 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.087641 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-envoy\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:22.087977 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.087954 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b23750-56cb-4895-80a8-8e54ae819f7a-istiod-ca-cert\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:22.088114 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.088097 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-workload-socket\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:22.088291 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.088274 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-data\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:22.088598 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.088565 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-podinfo\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:22.157572 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.066187 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-credential-socket\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:22.159198 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.124247 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b23750-56cb-4895-80a8-8e54ae819f7a-istiod-ca-cert\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:22.159401 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.124516 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-workload-socket\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:22.159533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.124755 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-data\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:22.159639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.065937 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-workload-certs\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:22.159729 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.127358 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-podinfo\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:22.159818 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.145076 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-envoy\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:22.159909 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.158853 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-token\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:22.183630 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.171641 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:22.183630 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.176973 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6vwk\" (UniqueName: \"kubernetes.io/projected/30b23750-56cb-4895-80a8-8e54ae819f7a-kube-api-access-n6vwk\") pod \"e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:22.183630 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.177703 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:22.341131 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.331931 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:22.762858 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.762824 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww"] Oct 30 18:18:22.764031 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:18:22.764004 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b23750_56cb_4895_80a8_8e54ae819f7a.slice/crio-d961ccfad98a23d4dd7ccaeaf941e44879fac714c046500a707d6dac0e49105c WatchSource:0}: Error finding container d961ccfad98a23d4dd7ccaeaf941e44879fac714c046500a707d6dac0e49105c: Status 404 returned error can't find the container with id d961ccfad98a23d4dd7ccaeaf941e44879fac714c046500a707d6dac0e49105c Oct 30 18:18:22.766753 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.766708 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:18:22.766838 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.766794 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:18:22.766838 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.766824 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:18:22.813051 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.813023 4603 patch_prober.go:29] interesting pod/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:18:22.813171 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.813070 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" podUID="a6ae148a-f57c-4430-b5ee-c17a7885bc78" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:18:22.903293 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.903251 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" event={"ID":"30b23750-56cb-4895-80a8-8e54ae819f7a","Type":"ContainerStarted","Data":"00c0b21e487800fbcf8c54e46f87001a571fa1f9f79e2f0dcbd0d555ec3b60cb"} Oct 30 18:18:22.903492 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.903314 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" event={"ID":"30b23750-56cb-4895-80a8-8e54ae819f7a","Type":"ContainerStarted","Data":"d961ccfad98a23d4dd7ccaeaf941e44879fac714c046500a707d6dac0e49105c"} Oct 30 18:18:22.948812 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:22.948760 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" podStartSLOduration=1.948746753 podStartE2EDuration="1.948746753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:22.948735336 +0000 UTC m=+1563.664305210" watchObservedRunningTime="2026-10-30 18:18:22.948746753 +0000 UTC m=+1563.664316627" Oct 30 18:18:23.183526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:23.182705 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:23.183526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:23.182853 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:18:23.183526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:23.182897 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" podUID="30b23750-56cb-4895-80a8-8e54ae819f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 18:18:23.678964 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:23.678932 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:24.182714 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:24.182690 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:24.839596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:24.839560 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:25.076337 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.076302 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:25.080131 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.078817 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:18:25.116357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.116107 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:25.831729 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.831705 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:18:25.847951 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.847928 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6ae148a-f57c-4430-b5ee-c17a7885bc78-kube-api-access-vj7bn\" (UniqueName: \"kubernetes.io/projected/a6ae148a-f57c-4430-b5ee-c17a7885bc78-kube-api-access-vj7bn\") pod \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " Oct 30 18:18:25.848074 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.847979 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-token\" (UniqueName: \"kubernetes.io/projected/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-token\") pod \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " Oct 30 18:18:25.848127 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.848091 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-envoy\") pod \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " Oct 30 18:18:25.848204 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.848190 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-credential-socket\") pod \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " Oct 30 18:18:25.848268 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.848221 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-workload-socket\") pod \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " Oct 30 18:18:25.848420 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.848398 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istiod-ca-cert\") pod \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " Oct 30 18:18:25.848534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.848443 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-credential-socket" pod "a6ae148a-f57c-4430-b5ee-c17a7885bc78" (UID: "a6ae148a-f57c-4430-b5ee-c17a7885bc78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:18:25.848534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.848517 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-podinfo\") pod \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " Oct 30 18:18:25.848656 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.848547 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-data\") pod \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " Oct 30 18:18:25.848656 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.848559 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-workload-socket" pod "a6ae148a-f57c-4430-b5ee-c17a7885bc78" (UID: "a6ae148a-f57c-4430-b5ee-c17a7885bc78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:18:25.848656 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.848612 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-workload-certs\") pod \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\" (UID: \"a6ae148a-f57c-4430-b5ee-c17a7885bc78\") " Oct 30 18:18:25.848966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.848806 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istiod-ca-cert" pod "a6ae148a-f57c-4430-b5ee-c17a7885bc78" (UID: "a6ae148a-f57c-4430-b5ee-c17a7885bc78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:25.849113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.849091 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-data" pod "a6ae148a-f57c-4430-b5ee-c17a7885bc78" (UID: "a6ae148a-f57c-4430-b5ee-c17a7885bc78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:18:25.849281 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.849259 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-workload-certs" pod "a6ae148a-f57c-4430-b5ee-c17a7885bc78" (UID: "a6ae148a-f57c-4430-b5ee-c17a7885bc78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:18:25.849372 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.849334 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:18:25.849372 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.849348 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:18:25.849372 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.849358 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:18:25.849372 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.849366 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:18:25.850613 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.850591 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-envoy" pod "a6ae148a-f57c-4430-b5ee-c17a7885bc78" (UID: "a6ae148a-f57c-4430-b5ee-c17a7885bc78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:18:25.850830 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.850802 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-token" pod "a6ae148a-f57c-4430-b5ee-c17a7885bc78" (UID: "a6ae148a-f57c-4430-b5ee-c17a7885bc78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:25.850918 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.850855 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-podinfo" pod "a6ae148a-f57c-4430-b5ee-c17a7885bc78" (UID: "a6ae148a-f57c-4430-b5ee-c17a7885bc78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:18:25.850918 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.850909 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ae148a-f57c-4430-b5ee-c17a7885bc78-kube-api-access-vj7bn" pod "a6ae148a-f57c-4430-b5ee-c17a7885bc78" (UID: "a6ae148a-f57c-4430-b5ee-c17a7885bc78"). InnerVolumeSpecName "kube-api-access-vj7bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:25.942392 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.942344 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:25.989587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.972913 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:18:25.989587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.972953 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:18:25.989587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.972968 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj7bn\" (UniqueName: \"kubernetes.io/projected/a6ae148a-f57c-4430-b5ee-c17a7885bc78-kube-api-access-vj7bn\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:18:25.989587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.972983 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:18:25.989587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:25.973004 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ae148a-f57c-4430-b5ee-c17a7885bc78-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:18:26.095597 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:26.088281 4603 generic.go:356] "Generic (PLEG): container finished" podID="a6ae148a-f57c-4430-b5ee-c17a7885bc78" containerID="a3c3af46e76453edd4cf01ebfe87e5920ca5532087d932c13c4f6f9858f6ab4c" exitCode=0 Oct 30 18:18:26.095597 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:26.088632 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" event={"ID":"a6ae148a-f57c-4430-b5ee-c17a7885bc78","Type":"ContainerDied","Data":"a3c3af46e76453edd4cf01ebfe87e5920ca5532087d932c13c4f6f9858f6ab4c"} Oct 30 18:18:26.095597 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:26.088721 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" event={"ID":"a6ae148a-f57c-4430-b5ee-c17a7885bc78","Type":"ContainerDied","Data":"87094005fe27f81cc08ff6e72d51f3a48b50eb46fe51713b8772a5fcff9200b1"} Oct 30 18:18:26.095597 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:26.088749 4603 scope.go:122] "RemoveContainer" containerID="a3c3af46e76453edd4cf01ebfe87e5920ca5532087d932c13c4f6f9858f6ab4c" Oct 30 18:18:26.095597 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:26.088984 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr" Oct 30 18:18:26.096275 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:26.096243 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:26.119733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:26.119710 4603 scope.go:122] "RemoveContainer" containerID="3030359edc4b632a8fd5a1e84a63364b37388b0be4da8fa8dc100d5a1fda6419" Oct 30 18:18:26.131563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:26.131541 4603 scope.go:122] "RemoveContainer" containerID="a3c3af46e76453edd4cf01ebfe87e5920ca5532087d932c13c4f6f9858f6ab4c" Oct 30 18:18:26.131902 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:18:26.131881 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3c3af46e76453edd4cf01ebfe87e5920ca5532087d932c13c4f6f9858f6ab4c\": container with ID starting with a3c3af46e76453edd4cf01ebfe87e5920ca5532087d932c13c4f6f9858f6ab4c not found: ID does not exist" containerID="a3c3af46e76453edd4cf01ebfe87e5920ca5532087d932c13c4f6f9858f6ab4c" Oct 30 18:18:26.131982 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:26.131911 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3c3af46e76453edd4cf01ebfe87e5920ca5532087d932c13c4f6f9858f6ab4c"} err="failed to get container status \"a3c3af46e76453edd4cf01ebfe87e5920ca5532087d932c13c4f6f9858f6ab4c\": rpc error: code = NotFound desc = could not find container \"a3c3af46e76453edd4cf01ebfe87e5920ca5532087d932c13c4f6f9858f6ab4c\": container with ID starting with a3c3af46e76453edd4cf01ebfe87e5920ca5532087d932c13c4f6f9858f6ab4c not found: ID does not exist" Oct 30 18:18:26.131982 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:26.131932 4603 scope.go:122] "RemoveContainer" containerID="3030359edc4b632a8fd5a1e84a63364b37388b0be4da8fa8dc100d5a1fda6419" Oct 30 18:18:26.132212 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:18:26.132192 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3030359edc4b632a8fd5a1e84a63364b37388b0be4da8fa8dc100d5a1fda6419\": container with ID starting with 3030359edc4b632a8fd5a1e84a63364b37388b0be4da8fa8dc100d5a1fda6419 not found: ID does not exist" containerID="3030359edc4b632a8fd5a1e84a63364b37388b0be4da8fa8dc100d5a1fda6419" Oct 30 18:18:26.132287 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:26.132226 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3030359edc4b632a8fd5a1e84a63364b37388b0be4da8fa8dc100d5a1fda6419"} err="failed to get container status \"3030359edc4b632a8fd5a1e84a63364b37388b0be4da8fa8dc100d5a1fda6419\": rpc error: code = NotFound desc = could not find container \"3030359edc4b632a8fd5a1e84a63364b37388b0be4da8fa8dc100d5a1fda6419\": container with ID starting with 3030359edc4b632a8fd5a1e84a63364b37388b0be4da8fa8dc100d5a1fda6419 not found: ID does not exist" Oct 30 18:18:26.154433 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:26.154400 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr"] Oct 30 18:18:26.162014 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:26.161993 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-da151d-openshift-default-569bdf86bc-p9lhr"] Oct 30 18:18:27.411307 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:27.411275 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6ae148a-f57c-4430-b5ee-c17a7885bc78" path="/var/lib/kubelet/pods/a6ae148a-f57c-4430-b5ee-c17a7885bc78/volumes" Oct 30 18:18:27.577491 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:27.577457 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:28.728552 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:28.728513 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:28.937525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:28.932122 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:29.199411 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:29.199382 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz"] Oct 30 18:18:29.200786 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:29.200747 4603 cpu_manager.go:414] "containerMap: removing container" podUID="a6ae148a-f57c-4430-b5ee-c17a7885bc78" containerName="istio-proxy" Oct 30 18:18:29.200978 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:29.200966 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6ae148a-f57c-4430-b5ee-c17a7885bc78" containerName="istio-proxy" Oct 30 18:18:29.201273 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:29.201260 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6ae148a-f57c-4430-b5ee-c17a7885bc78" containerName="istio-proxy" Oct 30 18:18:29.201371 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:29.201359 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6ae148a-f57c-4430-b5ee-c17a7885bc78" containerName="istio-proxy" Oct 30 18:18:29.232526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:29.225657 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz" Oct 30 18:18:29.232714 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:29.232631 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz"] Oct 30 18:18:29.255687 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:29.255622 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-064408\"" Oct 30 18:18:29.255944 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:29.255927 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-064408-dockercfg-bzqzh\"" Oct 30 18:18:29.291133 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:29.291097 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjkh5\" (UniqueName: \"kubernetes.io/projected/b5425f00-684b-4f8c-a9fb-7b7e534f66ed-kube-api-access-wjkh5\") pod \"payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz\" (UID: \"b5425f00-684b-4f8c-a9fb-7b7e534f66ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz" Oct 30 18:18:29.291425 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:29.291403 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5425f00-684b-4f8c-a9fb-7b7e534f66ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz\" (UID: \"b5425f00-684b-4f8c-a9fb-7b7e534f66ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz" Oct 30 18:18:29.392805 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:29.392771 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjkh5\" (UniqueName: \"kubernetes.io/projected/b5425f00-684b-4f8c-a9fb-7b7e534f66ed-kube-api-access-wjkh5\") pod \"payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz\" (UID: \"b5425f00-684b-4f8c-a9fb-7b7e534f66ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz" Oct 30 18:18:29.392995 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:29.392861 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5425f00-684b-4f8c-a9fb-7b7e534f66ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz\" (UID: \"b5425f00-684b-4f8c-a9fb-7b7e534f66ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz" Oct 30 18:18:29.393614 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:29.393586 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5425f00-684b-4f8c-a9fb-7b7e534f66ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz\" (UID: \"b5425f00-684b-4f8c-a9fb-7b7e534f66ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz" Oct 30 18:18:29.416420 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:29.416396 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjkh5\" (UniqueName: \"kubernetes.io/projected/b5425f00-684b-4f8c-a9fb-7b7e534f66ed-kube-api-access-wjkh5\") pod \"payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz\" (UID: \"b5425f00-684b-4f8c-a9fb-7b7e534f66ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz" Oct 30 18:18:29.556760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:29.556737 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz" Oct 30 18:18:29.824633 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:29.824562 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:29.926722 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:29.926443 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz"] Oct 30 18:18:30.195349 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:30.195264 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz" event={"ID":"b5425f00-684b-4f8c-a9fb-7b7e534f66ed","Type":"ContainerStarted","Data":"3bb4f52e441a48ef076240214b82908a4bc745fcc0f3437ee119f90ad63ad64b"} Oct 30 18:18:30.846117 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:30.846085 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:18:31.475074 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:31.475038 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:33.057528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:33.057284 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:33.273525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:33.269063 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz" event={"ID":"b5425f00-684b-4f8c-a9fb-7b7e534f66ed","Type":"ContainerStarted","Data":"0b8f80ac74b61a7ba131da0ed23029f798d9f7a6f5b8c1e05a71a83b0908df14"} Oct 30 18:18:33.276812 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:33.276779 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz" Oct 30 18:18:33.320704 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:33.320581 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz" podStartSLOduration=2.018862043 podStartE2EDuration="4.320558938s" totalImagesPullingTime="2.301696895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:33.319913729 +0000 UTC m=+1574.035483604" watchObservedRunningTime="2026-10-30 18:18:33.320558938 +0000 UTC m=+1574.036128817" Oct 30 18:18:33.854623 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:33.845088 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:34.136705 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:34.127384 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:34.509582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:34.507788 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:34.523562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:34.523518 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:34.525291 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:34.523781 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:18:34.607616 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:34.607573 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Oct 30 18:18:34.658631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:34.658576 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:34.824369 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:34.824335 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:34.847651 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:34.847603 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:18:34.864650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:34.864604 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:34.988858 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:34.988817 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:35.038529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:35.035625 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:35.518532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:35.514619 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:35.518532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:35.515858 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:35.518532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:35.516415 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:18:35.530539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:35.529787 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:18:35.530539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:35.530374 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:18:35.534527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:35.532574 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:35.534527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:35.533051 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:18:35.751546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:35.750979 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:35.751546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:35.751101 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:35.751835 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:35.751723 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:18:35.983600 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:35.863159 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:36.025528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:36.012599 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:18:36.025528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:36.013015 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:18:36.128533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:36.085610 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:36.128533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:36.108681 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:36.363527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:36.361335 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:36.416530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:36.415617 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:18:36.416530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:36.415715 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:18:36.416530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:36.415756 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:18:36.416530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:36.415792 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:18:36.556560 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:36.523264 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:36.556560 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:36.529983 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:36.561041 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:36.561005 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:36.810528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:36.787308 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:36.942962 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:36.942891 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:18:37.021659 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:37.021612 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:37.023197 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:37.021897 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:18:37.127708 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:37.123650 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:37.150726 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:37.150664 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Oct 30 18:18:37.446391 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:37.446285 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:37.740709 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:37.673598 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:37.740709 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:37.733153 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:18:37.740709 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:37.733466 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:18:37.905521 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:37.904583 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:38.026537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:38.025770 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:38.026537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:38.026193 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:18:38.076531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:38.061085 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:18:38.091156 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:38.091107 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:38.125627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:38.125580 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:38.451589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:38.444221 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:38.658530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:38.633489 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:38.678634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:38.678580 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:18:38.774265 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:38.774209 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:38.776935 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:38.774880 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:18:38.953612 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:38.932060 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:18:39.043589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:39.026612 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:39.043589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:39.027037 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:18:39.068655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:39.068607 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:39.130659 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:39.130605 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:39.684530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:39.683925 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:39.781537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:39.781093 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:39.807532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:39.803600 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:39.853530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:39.848129 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:39.853530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:39.848199 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" podUID="47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:39.898528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:39.898271 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:39.937589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:39.936627 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:40.034527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:40.032608 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:40.034527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:40.033009 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:18:40.076531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:40.075594 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:40.135631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:40.134631 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:18:40.135631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:40.135213 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:40.135631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:40.135256 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" podUID="e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:40.295667 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:40.295617 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Oct 30 18:18:40.296188 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:40.296161 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:40.296376 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:40.296355 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" podUID="db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:40.496517 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:40.496463 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:40.732274 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:40.730470 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:18:40.733250 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:40.733225 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:40.783228 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:40.783188 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:40.821745 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:40.817008 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268"] Oct 30 18:18:40.861726 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:40.817082 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:40.874140 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:40.874109 4603 cpu_manager.go:414] "containerMap: removing container" podUID="a6ae148a-f57c-4430-b5ee-c17a7885bc78" containerName="istio-proxy" Oct 30 18:18:40.886147 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:40.886117 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6ae148a-f57c-4430-b5ee-c17a7885bc78" containerName="istio-proxy" Oct 30 18:18:40.946790 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:40.946742 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:40.975754 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:40.975720 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268"] Oct 30 18:18:40.976144 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:40.976074 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268" Oct 30 18:18:41.007959 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.007872 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d951c4\"" Oct 30 18:18:41.092626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.022752 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d951c4-dockercfg-7nm7w\"" Oct 30 18:18:41.092880 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.035627 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.093376 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.035669 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:18:41.114530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.081088 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.125535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.123313 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq"] Oct 30 18:18:41.140793 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.140761 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq" Oct 30 18:18:41.147542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.142673 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.169719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.169679 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32a762e7-af53-435f-b602-f7ef85870b8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268\" (UID: \"32a762e7-af53-435f-b602-f7ef85870b8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268" Oct 30 18:18:41.215768 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.215729 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f048e3e7-0037-4d5f-93ae-eedf127e62a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq\" (UID: \"f048e3e7-0037-4d5f-93ae-eedf127e62a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq" Oct 30 18:18:41.244371 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.244334 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hscfd\" (UniqueName: \"kubernetes.io/projected/f048e3e7-0037-4d5f-93ae-eedf127e62a5-kube-api-access-hscfd\") pod \"payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq\" (UID: \"f048e3e7-0037-4d5f-93ae-eedf127e62a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq" Oct 30 18:18:41.244732 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.244711 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf2lx\" (UniqueName: \"kubernetes.io/projected/32a762e7-af53-435f-b602-f7ef85870b8e-kube-api-access-zf2lx\") pod \"payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268\" (UID: \"32a762e7-af53-435f-b602-f7ef85870b8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268" Oct 30 18:18:41.245110 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.177119 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq"] Oct 30 18:18:41.375558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.368122 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32a762e7-af53-435f-b602-f7ef85870b8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268\" (UID: \"32a762e7-af53-435f-b602-f7ef85870b8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268" Oct 30 18:18:41.408679 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.408644 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f048e3e7-0037-4d5f-93ae-eedf127e62a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq\" (UID: \"f048e3e7-0037-4d5f-93ae-eedf127e62a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq" Oct 30 18:18:41.461036 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.460992 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32a762e7-af53-435f-b602-f7ef85870b8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268\" (UID: \"32a762e7-af53-435f-b602-f7ef85870b8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268" Oct 30 18:18:41.493554 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.488360 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hscfd\" (UniqueName: \"kubernetes.io/projected/f048e3e7-0037-4d5f-93ae-eedf127e62a5-kube-api-access-hscfd\") pod \"payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq\" (UID: \"f048e3e7-0037-4d5f-93ae-eedf127e62a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq" Oct 30 18:18:41.493554 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.488491 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf2lx\" (UniqueName: \"kubernetes.io/projected/32a762e7-af53-435f-b602-f7ef85870b8e-kube-api-access-zf2lx\") pod \"payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268\" (UID: \"32a762e7-af53-435f-b602-f7ef85870b8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268" Oct 30 18:18:41.493554 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.489906 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f048e3e7-0037-4d5f-93ae-eedf127e62a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq\" (UID: \"f048e3e7-0037-4d5f-93ae-eedf127e62a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq" Oct 30 18:18:41.520198 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.520167 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:41.520473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.520437 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" podUID="c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.759615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.744973 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:18:41.759615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.746552 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf2lx\" (UniqueName: \"kubernetes.io/projected/32a762e7-af53-435f-b602-f7ef85870b8e-kube-api-access-zf2lx\") pod \"payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268\" (UID: \"32a762e7-af53-435f-b602-f7ef85870b8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268" Oct 30 18:18:41.850315 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.850264 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:18:41.875766 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.875724 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.919490 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.919452 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hscfd\" (UniqueName: \"kubernetes.io/projected/f048e3e7-0037-4d5f-93ae-eedf127e62a5-kube-api-access-hscfd\") pod \"payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq\" (UID: \"f048e3e7-0037-4d5f-93ae-eedf127e62a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq" Oct 30 18:18:41.938198 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.938167 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq" Oct 30 18:18:41.978588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:41.974472 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.032820 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:42.032790 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268" Oct 30 18:18:42.093636 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:42.093591 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:18:42.094844 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:42.094814 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.131099 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:42.131058 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.161663 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:42.161616 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.258102 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:42.257614 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:42.258102 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:42.257682 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" podUID="d5567046-aa24-421a-8339-fdd9cf56e62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.806540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:42.795086 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.837546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:42.836192 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:42.868256 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:42.868210 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.877614 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:42.877574 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:18:42.984737 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:42.983595 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:43.115582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:43.098634 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:43.115582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:43.100493 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:18:43.152529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:43.133632 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:43.175539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:43.162004 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:43.416719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:43.416616 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:43.425531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:43.416956 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" podUID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:43.807530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:43.795422 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:43.881212 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:43.869201 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:43.980605 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:43.979653 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:18:43.984105 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:43.984069 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:44.117916 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:44.117869 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:18:44.120271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:44.119684 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:44.159964 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:44.156112 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:44.165433 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:44.164587 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:44.180024 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:44.179965 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:44.180270 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:44.180229 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" podUID="22e0b280-2b65-4fad-8344-67049190d108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:44.490604 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:44.473659 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:44.490604 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:44.473724 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" podUID="5411300d-c1aa-4397-be3d-0ce869a1c1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:44.653464 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:44.653423 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:44.653957 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:44.653570 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:44.799537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:44.790094 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268"] Oct 30 18:18:44.819308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:44.819261 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:18:44.879941 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:44.879808 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:44.986658 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:44.986609 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:45.023884 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:45.005878 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:45.072590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:45.036382 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq"] Oct 30 18:18:45.133528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:45.122232 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:18:45.157410 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:45.136165 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:18:45.157410 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:45.156367 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:45.200719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:45.199738 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:18:45.503542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:45.499966 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Oct 30 18:18:45.587260 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:45.573288 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz" podUID="b5425f00-684b-4f8c-a9fb-7b7e534f66ed" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.101:9004: connect: connection refused" Oct 30 18:18:45.787236 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:45.787198 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268" event={"ID":"32a762e7-af53-435f-b602-f7ef85870b8e","Type":"ContainerStarted","Data":"08f001b4f41bfe994acf2069ff8f33234e96e078d5633958b4491b500acd9f43"} Oct 30 18:18:45.810778 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:45.810731 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq" event={"ID":"f048e3e7-0037-4d5f-93ae-eedf127e62a5","Type":"ContainerStarted","Data":"3339116fa6056e5ab023df22e9b2fe37aedbedb351cf831d2eedddc54fa1ba57"} Oct 30 18:18:45.847703 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:45.847619 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:45.888783 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:45.888719 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:45.993532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:45.988169 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:46.011793 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:46.011614 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:18:46.128618 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:46.124655 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:46.148567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:46.143631 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:46.162162 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:46.161645 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:18:46.200675 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:46.200631 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:46.302181 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:46.302094 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:46.845805 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:46.845775 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:18:46.847989 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:46.847958 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268" event={"ID":"32a762e7-af53-435f-b602-f7ef85870b8e","Type":"ContainerStarted","Data":"85326623dd831536af23ca9073de7e6f90a81bef200d252cd4859a0929a1eb0e"} Oct 30 18:18:46.850823 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:46.850806 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268" Oct 30 18:18:46.852262 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:46.852242 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq" event={"ID":"f048e3e7-0037-4d5f-93ae-eedf127e62a5","Type":"ContainerStarted","Data":"38bf3aabff8b2f5d035de2705deaa32ed8a51cdac9e3fa5c125e06d54c2fd1a6"} Oct 30 18:18:46.853292 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:46.853273 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq" Oct 30 18:18:46.853676 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:46.853655 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:46.868625 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:46.868608 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:18:46.886324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:46.885727 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:18:46.923529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:46.911869 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:18:46.963527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:46.951609 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:47.061044 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:47.058808 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:18:47.070357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:47.070307 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268" podStartSLOduration=7.070277206 podStartE2EDuration="7.070277206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:47.011776744 +0000 UTC m=+1587.727346631" watchObservedRunningTime="2026-10-30 18:18:47.070277206 +0000 UTC m=+1587.785847082" Oct 30 18:18:47.093570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:47.093541 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:18:47.166547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:47.166398 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq" podStartSLOduration=7.166379038 podStartE2EDuration="7.166379038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:47.165186669 +0000 UTC m=+1587.880756544" watchObservedRunningTime="2026-10-30 18:18:47.166379038 +0000 UTC m=+1587.881948895" Oct 30 18:18:47.169321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:47.169282 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:47.172318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:47.172190 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:18:47.194982 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:47.194955 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:18:47.885586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:47.885554 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69"] Oct 30 18:18:47.947491 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:47.946576 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw"] Oct 30 18:18:47.955493 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:47.955464 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:48.118529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:48.114332 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:48.266311 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:48.266169 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:50.253566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:50.253299 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:51.961525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:51.960180 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:52.083731 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:18:52.083694 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:18:52.886655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:52.886613 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69" podUID="050e2617-3912-482f-a910-c918b14efb6c" containerName="payload-processing" containerID="cri-o://fc4a5fb2ac7bff99a1e94ab22a28c400434d0d7f6fbf7398477a2e917a70911b" gracePeriod=25 Oct 30 18:18:52.947780 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:52.947742 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw" podUID="ecac0277-6449-4646-ab2c-918696deddf0" containerName="payload-pre-processing" containerID="cri-o://38b615730eaa08526bb7d05d2eb0c2028416d9016c4caf469c42ad8942da9eef" gracePeriod=25 Oct 30 18:18:53.125864 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:53.125827 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:53.126369 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:53.125902 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" podUID="18da6086-657d-4917-9fcb-c8de74e7f7b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:53.706340 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:53.704024 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw" Oct 30 18:18:53.724370 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:53.724299 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69" Oct 30 18:18:53.862837 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:53.862797 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecac0277-6449-4646-ab2c-918696deddf0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecac0277-6449-4646-ab2c-918696deddf0-plugins-config-volume\") pod \"ecac0277-6449-4646-ab2c-918696deddf0\" (UID: \"ecac0277-6449-4646-ab2c-918696deddf0\") " Oct 30 18:18:53.863025 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:53.862853 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/050e2617-3912-482f-a910-c918b14efb6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050e2617-3912-482f-a910-c918b14efb6c-plugins-config-volume\") pod \"050e2617-3912-482f-a910-c918b14efb6c\" (UID: \"050e2617-3912-482f-a910-c918b14efb6c\") " Oct 30 18:18:53.863025 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:53.862937 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecac0277-6449-4646-ab2c-918696deddf0-kube-api-access-6cb7f\" (UniqueName: \"kubernetes.io/projected/ecac0277-6449-4646-ab2c-918696deddf0-kube-api-access-6cb7f\") pod \"ecac0277-6449-4646-ab2c-918696deddf0\" (UID: \"ecac0277-6449-4646-ab2c-918696deddf0\") " Oct 30 18:18:53.863182 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:53.863065 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/050e2617-3912-482f-a910-c918b14efb6c-kube-api-access-9qmfb\" (UniqueName: \"kubernetes.io/projected/050e2617-3912-482f-a910-c918b14efb6c-kube-api-access-9qmfb\") pod \"050e2617-3912-482f-a910-c918b14efb6c\" (UID: \"050e2617-3912-482f-a910-c918b14efb6c\") " Oct 30 18:18:53.863342 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:53.863308 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecac0277-6449-4646-ab2c-918696deddf0-plugins-config-volume" pod "ecac0277-6449-4646-ab2c-918696deddf0" (UID: "ecac0277-6449-4646-ab2c-918696deddf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:53.863480 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:53.863363 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/050e2617-3912-482f-a910-c918b14efb6c-plugins-config-volume" pod "050e2617-3912-482f-a910-c918b14efb6c" (UID: "050e2617-3912-482f-a910-c918b14efb6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:53.863918 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:53.863899 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecac0277-6449-4646-ab2c-918696deddf0-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:18:53.863918 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:53.863918 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050e2617-3912-482f-a910-c918b14efb6c-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:18:53.865869 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:53.865844 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecac0277-6449-4646-ab2c-918696deddf0-kube-api-access-6cb7f" pod "ecac0277-6449-4646-ab2c-918696deddf0" (UID: "ecac0277-6449-4646-ab2c-918696deddf0"). InnerVolumeSpecName "kube-api-access-6cb7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:53.866001 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:53.865975 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050e2617-3912-482f-a910-c918b14efb6c-kube-api-access-9qmfb" pod "050e2617-3912-482f-a910-c918b14efb6c" (UID: "050e2617-3912-482f-a910-c918b14efb6c"). InnerVolumeSpecName "kube-api-access-9qmfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:53.965495 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:53.965458 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cb7f\" (UniqueName: \"kubernetes.io/projected/ecac0277-6449-4646-ab2c-918696deddf0-kube-api-access-6cb7f\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:18:53.965495 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:53.965514 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qmfb\" (UniqueName: \"kubernetes.io/projected/050e2617-3912-482f-a910-c918b14efb6c-kube-api-access-9qmfb\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:18:54.017588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:54.009464 4603 generic.go:356] "Generic (PLEG): container finished" podID="050e2617-3912-482f-a910-c918b14efb6c" containerID="fc4a5fb2ac7bff99a1e94ab22a28c400434d0d7f6fbf7398477a2e917a70911b" exitCode=0 Oct 30 18:18:54.017588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:54.009704 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69" event={"ID":"050e2617-3912-482f-a910-c918b14efb6c","Type":"ContainerDied","Data":"fc4a5fb2ac7bff99a1e94ab22a28c400434d0d7f6fbf7398477a2e917a70911b"} Oct 30 18:18:54.017588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:54.009791 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69" event={"ID":"050e2617-3912-482f-a910-c918b14efb6c","Type":"ContainerDied","Data":"f24bf05e8b8436d028a25c046316a87a34250b8dbcbf833dfa744260323d1ad3"} Oct 30 18:18:54.017588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:54.009824 4603 scope.go:122] "RemoveContainer" containerID="fc4a5fb2ac7bff99a1e94ab22a28c400434d0d7f6fbf7398477a2e917a70911b" Oct 30 18:18:54.017588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:54.009995 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69" Oct 30 18:18:54.022636 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:54.022610 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw" Oct 30 18:18:54.022903 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:54.022871 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw" event={"ID":"ecac0277-6449-4646-ab2c-918696deddf0","Type":"ContainerDied","Data":"38b615730eaa08526bb7d05d2eb0c2028416d9016c4caf469c42ad8942da9eef"} Oct 30 18:18:54.023957 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:54.023928 4603 generic.go:356] "Generic (PLEG): container finished" podID="ecac0277-6449-4646-ab2c-918696deddf0" containerID="38b615730eaa08526bb7d05d2eb0c2028416d9016c4caf469c42ad8942da9eef" exitCode=0 Oct 30 18:18:54.024069 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:54.024032 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw" event={"ID":"ecac0277-6449-4646-ab2c-918696deddf0","Type":"ContainerDied","Data":"8bc07cacaad7b135f36a21255f8abcd1c3219759b26d472c5a2fa4846e233664"} Oct 30 18:18:54.032693 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:54.032601 4603 scope.go:122] "RemoveContainer" containerID="fc4a5fb2ac7bff99a1e94ab22a28c400434d0d7f6fbf7398477a2e917a70911b" Oct 30 18:18:54.032929 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:18:54.032902 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4a5fb2ac7bff99a1e94ab22a28c400434d0d7f6fbf7398477a2e917a70911b\": container with ID starting with fc4a5fb2ac7bff99a1e94ab22a28c400434d0d7f6fbf7398477a2e917a70911b not found: ID does not exist" containerID="fc4a5fb2ac7bff99a1e94ab22a28c400434d0d7f6fbf7398477a2e917a70911b" Oct 30 18:18:54.032987 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:54.032944 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4a5fb2ac7bff99a1e94ab22a28c400434d0d7f6fbf7398477a2e917a70911b"} err="failed to get container status \"fc4a5fb2ac7bff99a1e94ab22a28c400434d0d7f6fbf7398477a2e917a70911b\": rpc error: code = NotFound desc = could not find container \"fc4a5fb2ac7bff99a1e94ab22a28c400434d0d7f6fbf7398477a2e917a70911b\": container with ID starting with fc4a5fb2ac7bff99a1e94ab22a28c400434d0d7f6fbf7398477a2e917a70911b not found: ID does not exist" Oct 30 18:18:54.032987 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:54.032967 4603 scope.go:122] "RemoveContainer" containerID="38b615730eaa08526bb7d05d2eb0c2028416d9016c4caf469c42ad8942da9eef" Oct 30 18:18:54.043180 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:54.043158 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:54.044386 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:54.044366 4603 scope.go:122] "RemoveContainer" containerID="38b615730eaa08526bb7d05d2eb0c2028416d9016c4caf469c42ad8942da9eef" Oct 30 18:18:54.044864 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:18:54.044842 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38b615730eaa08526bb7d05d2eb0c2028416d9016c4caf469c42ad8942da9eef\": container with ID starting with 38b615730eaa08526bb7d05d2eb0c2028416d9016c4caf469c42ad8942da9eef not found: ID does not exist" containerID="38b615730eaa08526bb7d05d2eb0c2028416d9016c4caf469c42ad8942da9eef" Oct 30 18:18:54.044952 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:54.044873 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b615730eaa08526bb7d05d2eb0c2028416d9016c4caf469c42ad8942da9eef"} err="failed to get container status \"38b615730eaa08526bb7d05d2eb0c2028416d9016c4caf469c42ad8942da9eef\": rpc error: code = NotFound desc = could not find container \"38b615730eaa08526bb7d05d2eb0c2028416d9016c4caf469c42ad8942da9eef\": container with ID starting with 38b615730eaa08526bb7d05d2eb0c2028416d9016c4caf469c42ad8942da9eef not found: ID does not exist" Oct 30 18:18:54.118652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:54.118610 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69"] Oct 30 18:18:54.149548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:54.146837 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-55db7948f4-tdt69"] Oct 30 18:18:54.244772 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:54.244738 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw"] Oct 30 18:18:54.247129 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:54.247096 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-54fd497bff-btklw"] Oct 30 18:18:55.425638 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:55.425602 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="050e2617-3912-482f-a910-c918b14efb6c" path="/var/lib/kubelet/pods/050e2617-3912-482f-a910-c918b14efb6c/volumes" Oct 30 18:18:55.426131 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:55.426069 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecac0277-6449-4646-ab2c-918696deddf0" path="/var/lib/kubelet/pods/ecac0277-6449-4646-ab2c-918696deddf0/volumes" Oct 30 18:18:55.426402 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:55.426389 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz" Oct 30 18:18:55.643066 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:55.643031 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:57.051367 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:57.051336 4603 generic.go:356] "Generic (PLEG): container finished" podID="d036618a-de13-4ab8-affb-7fc490084530" containerID="9936aa0093b73d9f0acdb8ae8c2a7ee8b9f9bf2b7b35147dcc208186c9b3bb01" exitCode=0 Oct 30 18:18:57.051918 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:57.051419 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" event={"ID":"d036618a-de13-4ab8-affb-7fc490084530","Type":"ContainerDied","Data":"9936aa0093b73d9f0acdb8ae8c2a7ee8b9f9bf2b7b35147dcc208186c9b3bb01"} Oct 30 18:18:57.052024 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:57.052004 4603 scope.go:122] "RemoveContainer" containerID="9936aa0093b73d9f0acdb8ae8c2a7ee8b9f9bf2b7b35147dcc208186c9b3bb01" Oct 30 18:18:57.053550 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:57.053530 4603 generic.go:356] "Generic (PLEG): container finished" podID="18da6086-657d-4917-9fcb-c8de74e7f7b4" containerID="3be733c96c1fbacee466c4189709eeeecf86becd478b651780424ad21f4e7a24" exitCode=0 Oct 30 18:18:57.053676 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:57.053612 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" event={"ID":"18da6086-657d-4917-9fcb-c8de74e7f7b4","Type":"ContainerDied","Data":"3be733c96c1fbacee466c4189709eeeecf86becd478b651780424ad21f4e7a24"} Oct 30 18:18:57.053948 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:57.053920 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:18:57.054054 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:57.054000 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:18:57.054054 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:57.054036 4603 scope.go:122] "RemoveContainer" containerID="3be733c96c1fbacee466c4189709eeeecf86becd478b651780424ad21f4e7a24" Oct 30 18:18:57.054164 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:57.054043 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:18:57.055757 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:57.055729 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:18:57.055865 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:57.055800 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:18:57.055865 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:57.055843 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:18:57.325254 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:57.325164 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:57.480527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:57.472988 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:57.835884 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:57.835850 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:58.112588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:58.108887 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" event={"ID":"18da6086-657d-4917-9fcb-c8de74e7f7b4","Type":"ContainerStarted","Data":"20f2d6ebcae7202fb4c21de11165d87f422f0074107df82dca9b64d52c516f41"} Oct 30 18:18:58.112588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:58.112266 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" event={"ID":"d036618a-de13-4ab8-affb-7fc490084530","Type":"ContainerStarted","Data":"dc7473b163dd47329282d0a44e6810910153f5d39f2bbaeabcd54f56dc27ff7d"} Oct 30 18:18:58.363589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:58.360815 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:18:58.367542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:58.366343 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:18:58.367542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:58.366401 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" podUID="18da6086-657d-4917-9fcb-c8de74e7f7b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 18:18:58.762580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:58.755550 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:18:58.762580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:58.755694 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:18:58.762580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:58.755739 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" podUID="d036618a-de13-4ab8-affb-7fc490084530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 18:18:58.929928 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:58.923690 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268" Oct 30 18:18:58.944528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:58.943854 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq" Oct 30 18:18:59.175213 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:59.165298 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:59.363934 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:59.363888 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:18:59.364115 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:59.363944 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" podUID="18da6086-657d-4917-9fcb-c8de74e7f7b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 18:18:59.754149 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:59.754112 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:18:59.754335 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:18:59.754185 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" podUID="d036618a-de13-4ab8-affb-7fc490084530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 18:19:00.357816 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:00.357771 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:19:00.358323 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:00.357835 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" podUID="18da6086-657d-4917-9fcb-c8de74e7f7b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 18:19:00.754573 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:00.754271 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:19:00.754573 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:00.754327 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" podUID="d036618a-de13-4ab8-affb-7fc490084530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 18:19:00.782241 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:00.782181 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:01.357734 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:01.357694 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:19:01.357934 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:01.357763 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" podUID="18da6086-657d-4917-9fcb-c8de74e7f7b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 18:19:01.754382 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:01.754297 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:19:01.754382 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:01.754370 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" podUID="d036618a-de13-4ab8-affb-7fc490084530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 18:19:01.943112 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:01.942995 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:02.124534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:02.106999 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:02.719601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:02.515028 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:19:02.719601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:02.515088 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" podUID="18da6086-657d-4917-9fcb-c8de74e7f7b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 18:19:02.719601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:02.523754 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr"] Oct 30 18:19:02.719601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:02.543916 4603 cpu_manager.go:414] "containerMap: removing container" podUID="ecac0277-6449-4646-ab2c-918696deddf0" containerName="payload-pre-processing" Oct 30 18:19:02.719601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:02.543947 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecac0277-6449-4646-ab2c-918696deddf0" containerName="payload-pre-processing" Oct 30 18:19:02.719601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:02.543980 4603 cpu_manager.go:414] "containerMap: removing container" podUID="050e2617-3912-482f-a910-c918b14efb6c" containerName="payload-processing" Oct 30 18:19:02.719601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:02.543991 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="050e2617-3912-482f-a910-c918b14efb6c" containerName="payload-processing" Oct 30 18:19:02.719601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:02.544169 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecac0277-6449-4646-ab2c-918696deddf0" containerName="payload-pre-processing" Oct 30 18:19:02.719601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:02.544186 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="050e2617-3912-482f-a910-c918b14efb6c" containerName="payload-processing" Oct 30 18:19:02.719601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:02.673266 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:02.859311 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:02.859274 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr"] Oct 30 18:19:02.859603 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:02.859586 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7"] Oct 30 18:19:02.893535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:02.892248 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:02.894120 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:02.893835 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:19:02.894120 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:02.893887 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" podUID="d036618a-de13-4ab8-affb-7fc490084530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 18:19:02.905639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:02.905610 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr" Oct 30 18:19:02.937748 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:02.936858 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7"] Oct 30 18:19:02.937986 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:02.936938 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7" Oct 30 18:19:02.952588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:02.952562 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c33ee4\"" Oct 30 18:19:02.952883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:02.952852 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c33ee4-dockercfg-xl6cc\"" Oct 30 18:19:03.049826 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:03.049790 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgzgq\" (UniqueName: \"kubernetes.io/projected/88e882e9-9a9c-4371-a61e-e5b8e707b2ac-kube-api-access-qgzgq\") pod \"payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7\" (UID: \"88e882e9-9a9c-4371-a61e-e5b8e707b2ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7" Oct 30 18:19:03.049992 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:03.049860 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e882e9-9a9c-4371-a61e-e5b8e707b2ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7\" (UID: \"88e882e9-9a9c-4371-a61e-e5b8e707b2ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7" Oct 30 18:19:03.050054 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:03.050011 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddxg8\" (UniqueName: \"kubernetes.io/projected/9b5e319e-7f09-43f1-8847-6c6a7cc73ade-kube-api-access-ddxg8\") pod \"payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr\" (UID: \"9b5e319e-7f09-43f1-8847-6c6a7cc73ade\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr" Oct 30 18:19:03.050106 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:03.050076 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5e319e-7f09-43f1-8847-6c6a7cc73ade-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr\" (UID: \"9b5e319e-7f09-43f1-8847-6c6a7cc73ade\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr" Oct 30 18:19:03.139986 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:03.100873 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:03.155528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:03.151661 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddxg8\" (UniqueName: \"kubernetes.io/projected/9b5e319e-7f09-43f1-8847-6c6a7cc73ade-kube-api-access-ddxg8\") pod \"payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr\" (UID: \"9b5e319e-7f09-43f1-8847-6c6a7cc73ade\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr" Oct 30 18:19:03.155528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:03.151742 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5e319e-7f09-43f1-8847-6c6a7cc73ade-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr\" (UID: \"9b5e319e-7f09-43f1-8847-6c6a7cc73ade\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr" Oct 30 18:19:03.155528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:03.151984 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgzgq\" (UniqueName: \"kubernetes.io/projected/88e882e9-9a9c-4371-a61e-e5b8e707b2ac-kube-api-access-qgzgq\") pod \"payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7\" (UID: \"88e882e9-9a9c-4371-a61e-e5b8e707b2ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7" Oct 30 18:19:03.155528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:03.152038 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e882e9-9a9c-4371-a61e-e5b8e707b2ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7\" (UID: \"88e882e9-9a9c-4371-a61e-e5b8e707b2ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7" Oct 30 18:19:03.155528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:03.153095 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e882e9-9a9c-4371-a61e-e5b8e707b2ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7\" (UID: \"88e882e9-9a9c-4371-a61e-e5b8e707b2ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7" Oct 30 18:19:03.155528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:03.154243 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5e319e-7f09-43f1-8847-6c6a7cc73ade-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr\" (UID: \"9b5e319e-7f09-43f1-8847-6c6a7cc73ade\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr" Oct 30 18:19:03.211979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:03.211947 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddxg8\" (UniqueName: \"kubernetes.io/projected/9b5e319e-7f09-43f1-8847-6c6a7cc73ade-kube-api-access-ddxg8\") pod \"payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr\" (UID: \"9b5e319e-7f09-43f1-8847-6c6a7cc73ade\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr" Oct 30 18:19:03.290571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:03.288535 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgzgq\" (UniqueName: \"kubernetes.io/projected/88e882e9-9a9c-4371-a61e-e5b8e707b2ac-kube-api-access-qgzgq\") pod \"payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7\" (UID: \"88e882e9-9a9c-4371-a61e-e5b8e707b2ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7" Oct 30 18:19:03.301565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:03.295856 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr" Oct 30 18:19:03.358526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:03.355642 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7" Oct 30 18:19:03.375860 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:03.374603 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:19:03.375860 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:03.374659 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" podUID="18da6086-657d-4917-9fcb-c8de74e7f7b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 18:19:03.739865 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:03.739834 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr"] Oct 30 18:19:03.753808 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:03.753779 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:19:03.753932 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:03.753835 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" podUID="d036618a-de13-4ab8-affb-7fc490084530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 18:19:03.847622 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:03.847573 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7"] Oct 30 18:19:03.987710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:03.987677 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:19:04.054405 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:03.987729 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" podUID="f7486973-71c0-4102-9f74-c814bf601ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:19:04.358054 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:04.358010 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:19:04.358254 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:04.358088 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" podUID="18da6086-657d-4917-9fcb-c8de74e7f7b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 18:19:04.595241 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:04.595149 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7" event={"ID":"88e882e9-9a9c-4371-a61e-e5b8e707b2ac","Type":"ContainerStarted","Data":"6d33c45dc536ec1fa5c29f9990d1e915a395627922662dce70d94530b80d6994"} Oct 30 18:19:04.595563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:04.595539 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7" event={"ID":"88e882e9-9a9c-4371-a61e-e5b8e707b2ac","Type":"ContainerStarted","Data":"949109d749b2b34f75ffe25305347d424521a1ddd958df52bdb09ab1e9527b47"} Oct 30 18:19:04.595990 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:04.595957 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7" Oct 30 18:19:04.605011 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:04.604969 4603 generic.go:356] "Generic (PLEG): container finished" podID="f7486973-71c0-4102-9f74-c814bf601ec5" containerID="c7b288f582bf7a4698353f293951d0b773c2be98ba32cb1f8abb2aa0de178bc3" exitCode=0 Oct 30 18:19:04.605136 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:04.605117 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" event={"ID":"f7486973-71c0-4102-9f74-c814bf601ec5","Type":"ContainerDied","Data":"c7b288f582bf7a4698353f293951d0b773c2be98ba32cb1f8abb2aa0de178bc3"} Oct 30 18:19:04.605647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:04.605624 4603 scope.go:122] "RemoveContainer" containerID="c7b288f582bf7a4698353f293951d0b773c2be98ba32cb1f8abb2aa0de178bc3" Oct 30 18:19:04.609100 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:04.609068 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:04.609263 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:04.609240 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:04.609435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:04.609366 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:04.610061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:04.610025 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr" event={"ID":"9b5e319e-7f09-43f1-8847-6c6a7cc73ade","Type":"ContainerStarted","Data":"58d4d941dc24ea86bf226f83075df6bc6da6bdc9b50d885b0e94189444334298"} Oct 30 18:19:04.612997 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:04.610751 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr" event={"ID":"9b5e319e-7f09-43f1-8847-6c6a7cc73ade","Type":"ContainerStarted","Data":"3f4c453b8c813b6b2fbc974214ae26b1a939ef060081ee9c7ea707f15bff4f15"} Oct 30 18:19:04.612997 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:04.611726 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr" Oct 30 18:19:04.643613 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:04.643553 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7" podStartSLOduration=2.643535196 podStartE2EDuration="2.643535196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:04.643144988 +0000 UTC m=+1605.358714863" watchObservedRunningTime="2026-10-30 18:19:04.643535196 +0000 UTC m=+1605.359105088" Oct 30 18:19:04.737535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:04.734587 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr" podStartSLOduration=2.734569872 podStartE2EDuration="2.734569872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:04.730662375 +0000 UTC m=+1605.446232250" watchObservedRunningTime="2026-10-30 18:19:04.734569872 +0000 UTC m=+1605.450139751" Oct 30 18:19:04.756459 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:04.756428 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:04.849577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:04.847954 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:19:04.852638 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:04.850478 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:19:04.856537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:04.854948 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:19:05.362187 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:05.362160 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:19:05.362634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:05.362615 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:19:05.363305 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:05.363285 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:19:05.451415 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:05.451384 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:05.630182 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:05.630086 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" event={"ID":"f7486973-71c0-4102-9f74-c814bf601ec5","Type":"ContainerStarted","Data":"6afbbbbda0c22724bb992044b34f8f4c8a558230f5a5ba229d5ed89d1adffa0a"} Oct 30 18:19:05.841293 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:05.841260 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx"] Oct 30 18:19:05.846345 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:05.846318 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:05.868548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:05.862793 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a8dc70-openshift-default-dockercfg-mn857\"" Oct 30 18:19:05.868699 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:05.868632 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx"] Oct 30 18:19:05.941199 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:05.941112 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:05.954160 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:05.953523 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-podinfo\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:05.954160 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:05.953589 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904c45e3-164e-45ee-b877-b8b565fbb7fd-istiod-ca-cert\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:05.954160 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:05.953645 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx2r2\" (UniqueName: \"kubernetes.io/projected/904c45e3-164e-45ee-b877-b8b565fbb7fd-kube-api-access-vx2r2\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:05.954160 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:05.953691 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-data\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:05.954160 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:05.953730 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-credential-socket\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:05.954160 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:05.953786 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-workload-certs\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:05.954160 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:05.953821 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-workload-socket\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:05.954160 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:05.953897 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-envoy\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:05.954160 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:05.953959 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-token\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:06.059773 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.059724 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx2r2\" (UniqueName: \"kubernetes.io/projected/904c45e3-164e-45ee-b877-b8b565fbb7fd-kube-api-access-vx2r2\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:06.060111 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.060084 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-data\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:06.060286 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.060269 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-credential-socket\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:06.060475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.060459 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-workload-certs\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:06.060647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.060632 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-workload-socket\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:06.060854 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.060837 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-envoy\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:06.061035 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.061008 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-token\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:06.061386 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.061368 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-podinfo\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:06.061624 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.061609 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904c45e3-164e-45ee-b877-b8b565fbb7fd-istiod-ca-cert\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:06.062755 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.062734 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904c45e3-164e-45ee-b877-b8b565fbb7fd-istiod-ca-cert\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:06.063236 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.063220 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-data\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:06.079439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.078869 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-workload-certs\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:06.079439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.079126 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-workload-socket\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:06.079891 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.079860 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-credential-socket\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:06.097537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.091957 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-podinfo\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:06.098130 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.098080 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-envoy\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:06.129189 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.129146 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-token\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:06.131195 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.131170 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx2r2\" (UniqueName: \"kubernetes.io/projected/904c45e3-164e-45ee-b877-b8b565fbb7fd-kube-api-access-vx2r2\") pod \"e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:06.136897 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.136872 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:06.162907 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.162878 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:06.561521 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.560382 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:19:06.561521 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.560587 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 18:19:06.561521 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.560650 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" podUID="f7486973-71c0-4102-9f74-c814bf601ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 18:19:06.580360 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.580321 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:06.830747 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.829118 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx"] Oct 30 18:19:06.834138 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.833322 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:06.834138 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.833417 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:06.834138 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:06.833463 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:07.543655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:07.543621 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 18:19:07.544080 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:07.543677 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" podUID="f7486973-71c0-4102-9f74-c814bf601ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 18:19:07.826741 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:07.826650 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" event={"ID":"904c45e3-164e-45ee-b877-b8b565fbb7fd","Type":"ContainerStarted","Data":"017d1e8d557fa8e9be5fb47c79ee543b2df3619eb2969c6a4350433fbba9abfa"} Oct 30 18:19:07.826892 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:07.826748 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" event={"ID":"904c45e3-164e-45ee-b877-b8b565fbb7fd","Type":"ContainerStarted","Data":"5dffc454bc20a3d2e164ffc8e8f27b0e9e12de073278df1b306ff106f21be0e6"} Oct 30 18:19:07.882804 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:07.882752 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" podStartSLOduration=2.882737701 podStartE2EDuration="2.882737701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:07.881045083 +0000 UTC m=+1608.596614959" watchObservedRunningTime="2026-10-30 18:19:07.882737701 +0000 UTC m=+1608.598307664" Oct 30 18:19:08.229597 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.229455 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:08.245652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.240978 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:08.258869 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.258834 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w"] Oct 30 18:19:08.267555 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.267528 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w"] Oct 30 18:19:08.267888 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.267825 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.609585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.609543 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ce9abc7-557e-4e1b-885c-410e0f51164c-istiod-ca-cert\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.649530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.642617 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-data\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.649530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.642753 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-token\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.649530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.642897 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-credential-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.649530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.642956 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlcgf\" (UniqueName: \"kubernetes.io/projected/0ce9abc7-557e-4e1b-885c-410e0f51164c-kube-api-access-tlcgf\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.649530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.642985 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-envoy\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.649530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.643041 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-workload-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.649530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.643069 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-podinfo\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.649530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.643117 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-workload-certs\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.649530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.634692 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 18:19:08.649530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.643314 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" podUID="f7486973-71c0-4102-9f74-c814bf601ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 18:19:08.745421 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.745384 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ce9abc7-557e-4e1b-885c-410e0f51164c-istiod-ca-cert\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.745630 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.745435 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-data\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.745630 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.745561 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-token\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.745765 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.745678 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-credential-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.745765 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.745727 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlcgf\" (UniqueName: \"kubernetes.io/projected/0ce9abc7-557e-4e1b-885c-410e0f51164c-kube-api-access-tlcgf\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.745765 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.745751 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-envoy\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.745933 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.745798 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-workload-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.745933 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.745826 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-podinfo\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.745933 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.745868 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-workload-certs\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.746110 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.745979 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-data\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.746165 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.746113 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-credential-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.746383 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.746305 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ce9abc7-557e-4e1b-885c-410e0f51164c-istiod-ca-cert\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.746383 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.746345 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-workload-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.746541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.746442 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-workload-certs\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.748460 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.748440 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-envoy\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.748924 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.748904 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-podinfo\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.773852 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.773821 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlcgf\" (UniqueName: \"kubernetes.io/projected/0ce9abc7-557e-4e1b-885c-410e0f51164c-kube-api-access-tlcgf\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.774643 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.774628 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-token\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.832978 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.832944 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6"] Oct 30 18:19:08.839934 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.839912 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:08.840715 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.840012 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:08.840715 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.840193 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:19:08.860912 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.860848 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6"] Oct 30 18:19:08.862656 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.862613 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-189cda-openshift-default-dockercfg-kv2k4\"" Oct 30 18:19:08.929364 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.929230 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:08.944034 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.943964 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:08.950672 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.949488 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-credential-socket\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:08.950672 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.949556 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-envoy\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:08.950672 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.949588 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-podinfo\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:08.950672 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.949631 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istiod-ca-cert\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:08.950672 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.949773 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldjkz\" (UniqueName: \"kubernetes.io/projected/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-kube-api-access-ldjkz\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:08.950672 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.949933 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-workload-socket\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:08.950672 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.949961 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-token\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:08.950672 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.950012 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-workload-certs\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:08.950672 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:08.950075 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-data\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:09.073526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.068039 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-workload-certs\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:09.073526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.068123 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-data\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:09.073526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.068468 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-credential-socket\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:09.073526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.068529 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-envoy\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:09.073526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.068557 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-podinfo\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:09.073526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.068621 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istiod-ca-cert\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:09.073526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.068772 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldjkz\" (UniqueName: \"kubernetes.io/projected/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-kube-api-access-ldjkz\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:09.073526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.068940 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-workload-socket\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:09.073526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.068967 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-token\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:09.073526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.069850 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-workload-certs\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:09.073526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.070111 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-data\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:09.073526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.070359 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-credential-socket\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:09.073526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.072891 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istiod-ca-cert\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:09.075576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.075553 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-workload-socket\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:09.116610 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.109180 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:09.301386 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.301311 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-token\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:09.325530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.324673 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-envoy\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:09.325530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.325083 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-podinfo\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:09.325530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.325310 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldjkz\" (UniqueName: \"kubernetes.io/projected/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-kube-api-access-ldjkz\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:09.467568 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.467444 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:09.545916 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.545885 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 18:19:09.546086 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.545936 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" podUID="f7486973-71c0-4102-9f74-c814bf601ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 18:19:09.564739 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.564711 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w"] Oct 30 18:19:09.578036 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.577852 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:09.578036 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.577932 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:09.578652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.578244 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:09.822834 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.822801 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6"] Oct 30 18:19:09.824936 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:19:09.824906 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a7164ea_a62e_4d5d_b26d_adcae50ff87c.slice/crio-9b636f28bfe45421a51cd006e25e6d5f3d742836f3c40257770a1b4bd56533a4 WatchSource:0}: Error finding container 9b636f28bfe45421a51cd006e25e6d5f3d742836f3c40257770a1b4bd56533a4: Status 404 returned error can't find the container with id 9b636f28bfe45421a51cd006e25e6d5f3d742836f3c40257770a1b4bd56533a4 Oct 30 18:19:09.827984 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.827951 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:09.828087 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.828016 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:09.828087 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.828047 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:09.849249 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.849219 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" event={"ID":"0ce9abc7-557e-4e1b-885c-410e0f51164c","Type":"ContainerStarted","Data":"6d94ee3965a0d533126fcfa61d335409ccf573b4ef5822e10a1395fa3a47deac"} Oct 30 18:19:09.849312 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.849288 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" event={"ID":"0ce9abc7-557e-4e1b-885c-410e0f51164c","Type":"ContainerStarted","Data":"f6e64ac89fe1849dd47ac17b33a136d052127f6c758e54b353feb647596536c5"} Oct 30 18:19:09.851875 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.851852 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" event={"ID":"4a7164ea-a62e-4d5d-b26d-adcae50ff87c","Type":"ContainerStarted","Data":"9b636f28bfe45421a51cd006e25e6d5f3d742836f3c40257770a1b4bd56533a4"} Oct 30 18:19:09.919734 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.919679 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" podStartSLOduration=1.919666001 podStartE2EDuration="1.919666001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:09.917856979 +0000 UTC m=+1610.633426855" watchObservedRunningTime="2026-10-30 18:19:09.919666001 +0000 UTC m=+1610.635235875" Oct 30 18:19:09.944135 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:09.944065 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:10.446165 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:10.446130 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:10.543064 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:10.543024 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 18:19:10.543249 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:10.543082 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" podUID="f7486973-71c0-4102-9f74-c814bf601ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 18:19:10.621527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:10.617258 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:10.861486 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:10.861452 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" event={"ID":"4a7164ea-a62e-4d5d-b26d-adcae50ff87c","Type":"ContainerStarted","Data":"9ccc70e0484185c438cedbb158cf6f81acc660fc8ef12e5eeb6ec63eea527416"} Oct 30 18:19:10.966616 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:10.964755 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:10.966616 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:10.965654 4603 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 18:19:10.966616 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:10.965715 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" podUID="0ce9abc7-557e-4e1b-885c-410e0f51164c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 18:19:10.982792 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:10.982730 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" podStartSLOduration=2.9827056929999998 podStartE2EDuration="2.982705693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:10.919814751 +0000 UTC m=+1611.635384653" watchObservedRunningTime="2026-10-30 18:19:10.982705693 +0000 UTC m=+1611.698275589" Oct 30 18:19:11.468003 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:11.467973 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:11.474037 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:11.474016 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:11.545528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:11.544886 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 18:19:11.545528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:11.544943 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" podUID="f7486973-71c0-4102-9f74-c814bf601ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 18:19:11.869295 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:11.869274 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:11.870316 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:11.870295 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:19:11.948643 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:11.948608 4603 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 18:19:11.948952 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:11.948925 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" podUID="0ce9abc7-557e-4e1b-885c-410e0f51164c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 18:19:12.266388 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:12.266309 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:12.435464 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:12.435434 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:12.848063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:12.848013 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:19:12.851468 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:12.851445 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:19:12.855491 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:12.855468 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:19:12.944928 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:12.944896 4603 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 18:19:12.945304 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:12.944947 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" podUID="0ce9abc7-557e-4e1b-885c-410e0f51164c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 18:19:13.427111 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:13.427081 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:13.572065 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:13.572023 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:13.807853 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:13.807792 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:13.948040 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:13.948007 4603 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 18:19:13.948653 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:13.948625 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" podUID="0ce9abc7-557e-4e1b-885c-410e0f51164c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 18:19:14.944942 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:14.944913 4603 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 18:19:14.945095 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:14.944961 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" podUID="0ce9abc7-557e-4e1b-885c-410e0f51164c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 18:19:15.179266 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:15.179179 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:15.346900 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:15.346870 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:15.556680 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:15.546774 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:15.657400 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:15.656951 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7" Oct 30 18:19:15.737328 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:15.737289 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:15.992417 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:15.992323 4603 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 18:19:16.005665 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:16.005618 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" podUID="0ce9abc7-557e-4e1b-885c-410e0f51164c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 18:19:16.763431 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:16.763389 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:16.820262 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:16.820234 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr" Oct 30 18:19:16.946956 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:16.946917 4603 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 18:19:16.947247 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:16.947215 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" podUID="0ce9abc7-557e-4e1b-885c-410e0f51164c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 18:19:17.350721 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:17.350691 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:17.634528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:17.634418 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:17.963952 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:17.963861 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:18.085588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:18.033578 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:18.085588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:18.033726 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:19.160540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:19.160466 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:20.095483 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.095456 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:20.491530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.491422 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb"] Oct 30 18:19:20.497811 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.497789 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.522817 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.522793 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb"] Oct 30 18:19:20.528494 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.528471 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-069812-openshift-default-dockercfg-cfwtt\"" Oct 30 18:19:20.551180 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.551144 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-workload-certs\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.551714 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.551688 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-workload-socket\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.551861 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.551834 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-credential-socket\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.551924 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.551882 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq5bv\" (UniqueName: \"kubernetes.io/projected/3472e0a8-d76d-45af-a9f4-cae9b46712b6-kube-api-access-wq5bv\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.551984 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.551940 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-podinfo\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.552052 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.552016 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-envoy\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.552118 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.552103 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-data\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.552231 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.552216 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-token\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.552295 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.552246 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istiod-ca-cert\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.653447 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.653415 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-envoy\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.653667 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.653488 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-data\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.653667 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.653621 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-token\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.653667 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.653641 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istiod-ca-cert\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.654250 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.653680 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-workload-certs\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.654250 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.653774 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-workload-socket\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.654250 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.653851 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-credential-socket\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.654250 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.653883 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq5bv\" (UniqueName: \"kubernetes.io/projected/3472e0a8-d76d-45af-a9f4-cae9b46712b6-kube-api-access-wq5bv\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.654250 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.653943 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-podinfo\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.654250 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.653943 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-data\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.654250 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.654148 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-workload-certs\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.654495 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.654266 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-credential-socket\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.654495 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.654442 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istiod-ca-cert\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.654495 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.654471 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-workload-socket\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.655967 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.655947 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-envoy\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.656597 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.656577 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-podinfo\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.767572 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.766081 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-token\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.786560 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.768170 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq5bv\" (UniqueName: \"kubernetes.io/projected/3472e0a8-d76d-45af-a9f4-cae9b46712b6-kube-api-access-wq5bv\") pod \"e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.786560 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.786152 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:20.825524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.824801 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:20.958532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:20.946177 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:21.121235 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:21.120305 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268"] Oct 30 18:19:21.151645 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:21.151613 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq"] Oct 30 18:19:21.210517 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:21.210473 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:21.445872 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:21.444921 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:21.476042 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:21.475734 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:21.476042 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:21.475835 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:21.476042 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:21.475874 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:21.504255 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:21.504218 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb"] Oct 30 18:19:21.640446 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:21.640349 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:22.138776 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:19:22.138735 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:19:22.196647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:22.196612 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" event={"ID":"3472e0a8-d76d-45af-a9f4-cae9b46712b6","Type":"ContainerStarted","Data":"914b115695314e21fb1df670d9d1d9d69a1c192160a66d27a4fa26c6ae1dafbf"} Oct 30 18:19:22.196856 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:22.196665 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" event={"ID":"3472e0a8-d76d-45af-a9f4-cae9b46712b6","Type":"ContainerStarted","Data":"60d4b160b41d53a21809cddae5f7d767a64d26564e3ad6588bffc7a379ca1691"} Oct 30 18:19:22.254467 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:22.254367 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" podStartSLOduration=2.254347201 podStartE2EDuration="2.254347201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:22.25168565 +0000 UTC m=+1622.967255561" watchObservedRunningTime="2026-10-30 18:19:22.254347201 +0000 UTC m=+1622.969917079" Oct 30 18:19:22.825280 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:22.825254 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:22.830824 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:22.830802 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:23.203638 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:23.203548 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:23.204411 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:23.204391 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:19:23.467982 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:23.467889 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:23.904714 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:23.904666 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:24.041656 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:24.041593 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:24.210618 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:24.210527 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq"] Oct 30 18:19:24.210870 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:24.210838 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" podUID="02a7d3cd-7df0-41c6-9063-f2caf0e934d4" containerName="istio-proxy" containerID="cri-o://a27e5ec44600057d403fb860e9675144e27c797945be7f4cb1b40c78d5ec00b0" gracePeriod=30 Oct 30 18:19:24.227103 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:24.227072 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w"] Oct 30 18:19:24.227280 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:24.227259 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" podUID="0ce9abc7-557e-4e1b-885c-410e0f51164c" containerName="istio-proxy" containerID="cri-o://6d94ee3965a0d533126fcfa61d335409ccf573b4ef5822e10a1395fa3a47deac" gracePeriod=30 Oct 30 18:19:24.272111 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:24.272074 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:25.067823 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:25.067791 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:25.300534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:25.292049 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:25.585854 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:25.585823 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:25.981186 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:25.981099 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:26.123332 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:26.123269 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268" podUID="32a762e7-af53-435f-b602-f7ef85870b8e" containerName="payload-processing" containerID="cri-o://85326623dd831536af23ca9073de7e6f90a81bef200d252cd4859a0929a1eb0e" gracePeriod=25 Oct 30 18:19:26.152401 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:26.152371 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq" podUID="f048e3e7-0037-4d5f-93ae-eedf127e62a5" containerName="payload-pre-processing" containerID="cri-o://38bf3aabff8b2f5d035de2705deaa32ed8a51cdac9e3fa5c125e06d54c2fd1a6" gracePeriod=25 Oct 30 18:19:26.246684 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:26.246621 4603 generic.go:356] "Generic (PLEG): container finished" podID="32a762e7-af53-435f-b602-f7ef85870b8e" containerID="85326623dd831536af23ca9073de7e6f90a81bef200d252cd4859a0929a1eb0e" exitCode=0 Oct 30 18:19:26.246813 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:26.246690 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268" event={"ID":"32a762e7-af53-435f-b602-f7ef85870b8e","Type":"ContainerDied","Data":"85326623dd831536af23ca9073de7e6f90a81bef200d252cd4859a0929a1eb0e"} Oct 30 18:19:26.440738 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:26.432292 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268" Oct 30 18:19:26.492155 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:26.492123 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32a762e7-af53-435f-b602-f7ef85870b8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32a762e7-af53-435f-b602-f7ef85870b8e-plugins-config-volume\") pod \"32a762e7-af53-435f-b602-f7ef85870b8e\" (UID: \"32a762e7-af53-435f-b602-f7ef85870b8e\") " Oct 30 18:19:26.492401 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:26.492378 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a762e7-af53-435f-b602-f7ef85870b8e-kube-api-access-zf2lx\" (UniqueName: \"kubernetes.io/projected/32a762e7-af53-435f-b602-f7ef85870b8e-kube-api-access-zf2lx\") pod \"32a762e7-af53-435f-b602-f7ef85870b8e\" (UID: \"32a762e7-af53-435f-b602-f7ef85870b8e\") " Oct 30 18:19:26.492543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:26.492496 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32a762e7-af53-435f-b602-f7ef85870b8e-plugins-config-volume" pod "32a762e7-af53-435f-b602-f7ef85870b8e" (UID: "32a762e7-af53-435f-b602-f7ef85870b8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:26.493941 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:26.493477 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32a762e7-af53-435f-b602-f7ef85870b8e-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:19:26.496439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:26.496392 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a762e7-af53-435f-b602-f7ef85870b8e-kube-api-access-zf2lx" pod "32a762e7-af53-435f-b602-f7ef85870b8e" (UID: "32a762e7-af53-435f-b602-f7ef85870b8e"). InnerVolumeSpecName "kube-api-access-zf2lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:26.522458 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:26.522438 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq" Oct 30 18:19:26.594446 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:26.594418 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f048e3e7-0037-4d5f-93ae-eedf127e62a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f048e3e7-0037-4d5f-93ae-eedf127e62a5-plugins-config-volume\") pod \"f048e3e7-0037-4d5f-93ae-eedf127e62a5\" (UID: \"f048e3e7-0037-4d5f-93ae-eedf127e62a5\") " Oct 30 18:19:26.594699 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:26.594684 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f048e3e7-0037-4d5f-93ae-eedf127e62a5-kube-api-access-hscfd\" (UniqueName: \"kubernetes.io/projected/f048e3e7-0037-4d5f-93ae-eedf127e62a5-kube-api-access-hscfd\") pod \"f048e3e7-0037-4d5f-93ae-eedf127e62a5\" (UID: \"f048e3e7-0037-4d5f-93ae-eedf127e62a5\") " Oct 30 18:19:26.594837 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:26.594810 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f048e3e7-0037-4d5f-93ae-eedf127e62a5-plugins-config-volume" pod "f048e3e7-0037-4d5f-93ae-eedf127e62a5" (UID: "f048e3e7-0037-4d5f-93ae-eedf127e62a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:26.595168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:26.595150 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f048e3e7-0037-4d5f-93ae-eedf127e62a5-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:19:26.595168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:26.595168 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf2lx\" (UniqueName: \"kubernetes.io/projected/32a762e7-af53-435f-b602-f7ef85870b8e-kube-api-access-zf2lx\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:19:26.596838 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:26.596817 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f048e3e7-0037-4d5f-93ae-eedf127e62a5-kube-api-access-hscfd" pod "f048e3e7-0037-4d5f-93ae-eedf127e62a5" (UID: "f048e3e7-0037-4d5f-93ae-eedf127e62a5"). InnerVolumeSpecName "kube-api-access-hscfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:26.696931 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:26.696907 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hscfd\" (UniqueName: \"kubernetes.io/projected/f048e3e7-0037-4d5f-93ae-eedf127e62a5-kube-api-access-hscfd\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:19:27.069155 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:27.069115 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:27.254929 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:27.254902 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268" Oct 30 18:19:27.255367 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:27.254902 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268" event={"ID":"32a762e7-af53-435f-b602-f7ef85870b8e","Type":"ContainerDied","Data":"08f001b4f41bfe994acf2069ff8f33234e96e078d5633958b4491b500acd9f43"} Oct 30 18:19:27.255367 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:27.255032 4603 scope.go:122] "RemoveContainer" containerID="85326623dd831536af23ca9073de7e6f90a81bef200d252cd4859a0929a1eb0e" Oct 30 18:19:27.256566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:27.256540 4603 generic.go:356] "Generic (PLEG): container finished" podID="f048e3e7-0037-4d5f-93ae-eedf127e62a5" containerID="38bf3aabff8b2f5d035de2705deaa32ed8a51cdac9e3fa5c125e06d54c2fd1a6" exitCode=0 Oct 30 18:19:27.256706 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:27.256606 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq" Oct 30 18:19:27.256706 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:27.256658 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq" event={"ID":"f048e3e7-0037-4d5f-93ae-eedf127e62a5","Type":"ContainerDied","Data":"38bf3aabff8b2f5d035de2705deaa32ed8a51cdac9e3fa5c125e06d54c2fd1a6"} Oct 30 18:19:27.256795 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:27.256715 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq" event={"ID":"f048e3e7-0037-4d5f-93ae-eedf127e62a5","Type":"ContainerDied","Data":"3339116fa6056e5ab023df22e9b2fe37aedbedb351cf831d2eedddc54fa1ba57"} Oct 30 18:19:27.265989 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:27.265973 4603 scope.go:122] "RemoveContainer" containerID="38bf3aabff8b2f5d035de2705deaa32ed8a51cdac9e3fa5c125e06d54c2fd1a6" Oct 30 18:19:27.275168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:27.275151 4603 scope.go:122] "RemoveContainer" containerID="38bf3aabff8b2f5d035de2705deaa32ed8a51cdac9e3fa5c125e06d54c2fd1a6" Oct 30 18:19:27.275428 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:19:27.275406 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38bf3aabff8b2f5d035de2705deaa32ed8a51cdac9e3fa5c125e06d54c2fd1a6\": container with ID starting with 38bf3aabff8b2f5d035de2705deaa32ed8a51cdac9e3fa5c125e06d54c2fd1a6 not found: ID does not exist" containerID="38bf3aabff8b2f5d035de2705deaa32ed8a51cdac9e3fa5c125e06d54c2fd1a6" Oct 30 18:19:27.275491 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:27.275437 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38bf3aabff8b2f5d035de2705deaa32ed8a51cdac9e3fa5c125e06d54c2fd1a6"} err="failed to get container status \"38bf3aabff8b2f5d035de2705deaa32ed8a51cdac9e3fa5c125e06d54c2fd1a6\": rpc error: code = NotFound desc = could not find container \"38bf3aabff8b2f5d035de2705deaa32ed8a51cdac9e3fa5c125e06d54c2fd1a6\": container with ID starting with 38bf3aabff8b2f5d035de2705deaa32ed8a51cdac9e3fa5c125e06d54c2fd1a6 not found: ID does not exist" Oct 30 18:19:27.306267 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:27.306242 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268"] Oct 30 18:19:27.317580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:27.317562 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d951c4-fc6df759b-5n268"] Oct 30 18:19:27.352310 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:27.352254 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq"] Oct 30 18:19:27.355878 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:27.355859 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-7d498c4b75-rc9nq"] Oct 30 18:19:27.410903 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:27.410882 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32a762e7-af53-435f-b602-f7ef85870b8e" path="/var/lib/kubelet/pods/32a762e7-af53-435f-b602-f7ef85870b8e/volumes" Oct 30 18:19:27.411303 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:27.411290 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f048e3e7-0037-4d5f-93ae-eedf127e62a5" path="/var/lib/kubelet/pods/f048e3e7-0037-4d5f-93ae-eedf127e62a5/volumes" Oct 30 18:19:29.318399 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.308542 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:29.477193 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.477160 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:29.639757 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.639728 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:19:29.663365 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.663332 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:29.671019 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.670997 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-workload-certs\") pod \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " Oct 30 18:19:29.671134 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.671037 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-credential-socket\") pod \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " Oct 30 18:19:29.671134 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.671128 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-kube-api-access-gx4xj\" (UniqueName: \"kubernetes.io/projected/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-kube-api-access-gx4xj\") pod \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " Oct 30 18:19:29.671263 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.671212 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istiod-ca-cert\") pod \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " Oct 30 18:19:29.671323 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.671269 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-workload-certs" pod "02a7d3cd-7df0-41c6-9063-f2caf0e934d4" (UID: "02a7d3cd-7df0-41c6-9063-f2caf0e934d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.671323 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.671285 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-podinfo\") pod \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " Oct 30 18:19:29.671434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.671334 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-data\") pod \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " Oct 30 18:19:29.671434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.671368 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-credential-socket" pod "02a7d3cd-7df0-41c6-9063-f2caf0e934d4" (UID: "02a7d3cd-7df0-41c6-9063-f2caf0e934d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.671434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.671382 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-token\" (UniqueName: \"kubernetes.io/projected/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-token\") pod \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " Oct 30 18:19:29.671434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.671429 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-workload-socket\") pod \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " Oct 30 18:19:29.671761 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.671464 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-envoy\") pod \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\" (UID: \"02a7d3cd-7df0-41c6-9063-f2caf0e934d4\") " Oct 30 18:19:29.671761 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.671644 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istiod-ca-cert" pod "02a7d3cd-7df0-41c6-9063-f2caf0e934d4" (UID: "02a7d3cd-7df0-41c6-9063-f2caf0e934d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:29.671866 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.671806 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-workload-socket" pod "02a7d3cd-7df0-41c6-9063-f2caf0e934d4" (UID: "02a7d3cd-7df0-41c6-9063-f2caf0e934d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.671962 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.671944 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-data" pod "02a7d3cd-7df0-41c6-9063-f2caf0e934d4" (UID: "02a7d3cd-7df0-41c6-9063-f2caf0e934d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.672519 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.672483 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.672680 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.672643 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.672826 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.672810 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.672932 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.672919 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.673026 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.673005 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.675247 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.675105 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-podinfo" pod "02a7d3cd-7df0-41c6-9063-f2caf0e934d4" (UID: "02a7d3cd-7df0-41c6-9063-f2caf0e934d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:19:29.675247 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.675209 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-envoy" pod "02a7d3cd-7df0-41c6-9063-f2caf0e934d4" (UID: "02a7d3cd-7df0-41c6-9063-f2caf0e934d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.676933 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.676912 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-token" pod "02a7d3cd-7df0-41c6-9063-f2caf0e934d4" (UID: "02a7d3cd-7df0-41c6-9063-f2caf0e934d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:29.677796 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.677762 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-kube-api-access-gx4xj" pod "02a7d3cd-7df0-41c6-9063-f2caf0e934d4" (UID: "02a7d3cd-7df0-41c6-9063-f2caf0e934d4"). InnerVolumeSpecName "kube-api-access-gx4xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:29.773768 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.773738 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-token\" (UniqueName: \"kubernetes.io/projected/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-token\") pod \"0ce9abc7-557e-4e1b-885c-410e0f51164c\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " Oct 30 18:19:29.773942 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.773804 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-credential-socket\") pod \"0ce9abc7-557e-4e1b-885c-410e0f51164c\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " Oct 30 18:19:29.774008 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.773957 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-workload-certs\") pod \"0ce9abc7-557e-4e1b-885c-410e0f51164c\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " Oct 30 18:19:29.774008 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.773996 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ce9abc7-557e-4e1b-885c-410e0f51164c-kube-api-access-tlcgf\" (UniqueName: \"kubernetes.io/projected/0ce9abc7-557e-4e1b-885c-410e0f51164c-kube-api-access-tlcgf\") pod \"0ce9abc7-557e-4e1b-885c-410e0f51164c\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " Oct 30 18:19:29.774130 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.774099 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-data\") pod \"0ce9abc7-557e-4e1b-885c-410e0f51164c\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " Oct 30 18:19:29.774187 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.774131 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ce9abc7-557e-4e1b-885c-410e0f51164c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ce9abc7-557e-4e1b-885c-410e0f51164c-istiod-ca-cert\") pod \"0ce9abc7-557e-4e1b-885c-410e0f51164c\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " Oct 30 18:19:29.774187 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.774172 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-credential-socket" pod "0ce9abc7-557e-4e1b-885c-410e0f51164c" (UID: "0ce9abc7-557e-4e1b-885c-410e0f51164c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.774299 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.774191 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-workload-socket\") pod \"0ce9abc7-557e-4e1b-885c-410e0f51164c\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " Oct 30 18:19:29.774353 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.774334 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-envoy\") pod \"0ce9abc7-557e-4e1b-885c-410e0f51164c\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " Oct 30 18:19:29.774407 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.774367 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-podinfo\") pod \"0ce9abc7-557e-4e1b-885c-410e0f51164c\" (UID: \"0ce9abc7-557e-4e1b-885c-410e0f51164c\") " Oct 30 18:19:29.775034 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.774364 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-data" pod "0ce9abc7-557e-4e1b-885c-410e0f51164c" (UID: "0ce9abc7-557e-4e1b-885c-410e0f51164c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.775034 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.774525 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ce9abc7-557e-4e1b-885c-410e0f51164c-istiod-ca-cert" pod "0ce9abc7-557e-4e1b-885c-410e0f51164c" (UID: "0ce9abc7-557e-4e1b-885c-410e0f51164c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:29.775034 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.774674 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-workload-certs" pod "0ce9abc7-557e-4e1b-885c-410e0f51164c" (UID: "0ce9abc7-557e-4e1b-885c-410e0f51164c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.775034 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.774958 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-workload-socket" pod "0ce9abc7-557e-4e1b-885c-410e0f51164c" (UID: "0ce9abc7-557e-4e1b-885c-410e0f51164c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.775333 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.775190 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.775333 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.775213 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.775333 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.775228 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx4xj\" (UniqueName: \"kubernetes.io/projected/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-kube-api-access-gx4xj\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.775333 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.775242 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.775333 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.775255 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.775333 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.775268 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ce9abc7-557e-4e1b-885c-410e0f51164c-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.775333 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.775280 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.775333 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.775293 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.775333 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.775306 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02a7d3cd-7df0-41c6-9063-f2caf0e934d4-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.777111 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.777081 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce9abc7-557e-4e1b-885c-410e0f51164c-kube-api-access-tlcgf" pod "0ce9abc7-557e-4e1b-885c-410e0f51164c" (UID: "0ce9abc7-557e-4e1b-885c-410e0f51164c"). InnerVolumeSpecName "kube-api-access-tlcgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:29.777386 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.777360 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-podinfo" pod "0ce9abc7-557e-4e1b-885c-410e0f51164c" (UID: "0ce9abc7-557e-4e1b-885c-410e0f51164c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:19:29.777496 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.777474 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-envoy" pod "0ce9abc7-557e-4e1b-885c-410e0f51164c" (UID: "0ce9abc7-557e-4e1b-885c-410e0f51164c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.777800 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.777777 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-token" pod "0ce9abc7-557e-4e1b-885c-410e0f51164c" (UID: "0ce9abc7-557e-4e1b-885c-410e0f51164c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:29.876295 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.876269 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.876295 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.876295 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.876480 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.876304 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlcgf\" (UniqueName: \"kubernetes.io/projected/0ce9abc7-557e-4e1b-885c-410e0f51164c-kube-api-access-tlcgf\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.876480 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:29.876312 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ce9abc7-557e-4e1b-885c-410e0f51164c-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:19:30.279892 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:30.279858 4603 generic.go:356] "Generic (PLEG): container finished" podID="0ce9abc7-557e-4e1b-885c-410e0f51164c" containerID="6d94ee3965a0d533126fcfa61d335409ccf573b4ef5822e10a1395fa3a47deac" exitCode=0 Oct 30 18:19:30.280075 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:30.279921 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" Oct 30 18:19:30.280075 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:30.279963 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" event={"ID":"0ce9abc7-557e-4e1b-885c-410e0f51164c","Type":"ContainerDied","Data":"6d94ee3965a0d533126fcfa61d335409ccf573b4ef5822e10a1395fa3a47deac"} Oct 30 18:19:30.280075 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:30.280038 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w" event={"ID":"0ce9abc7-557e-4e1b-885c-410e0f51164c","Type":"ContainerDied","Data":"f6e64ac89fe1849dd47ac17b33a136d052127f6c758e54b353feb647596536c5"} Oct 30 18:19:30.280227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:30.280077 4603 scope.go:122] "RemoveContainer" containerID="6d94ee3965a0d533126fcfa61d335409ccf573b4ef5822e10a1395fa3a47deac" Oct 30 18:19:30.282058 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:30.282023 4603 generic.go:356] "Generic (PLEG): container finished" podID="02a7d3cd-7df0-41c6-9063-f2caf0e934d4" containerID="a27e5ec44600057d403fb860e9675144e27c797945be7f4cb1b40c78d5ec00b0" exitCode=0 Oct 30 18:19:30.282182 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:30.282067 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" Oct 30 18:19:30.282182 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:30.282066 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" event={"ID":"02a7d3cd-7df0-41c6-9063-f2caf0e934d4","Type":"ContainerDied","Data":"a27e5ec44600057d403fb860e9675144e27c797945be7f4cb1b40c78d5ec00b0"} Oct 30 18:19:30.282182 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:30.282120 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq" event={"ID":"02a7d3cd-7df0-41c6-9063-f2caf0e934d4","Type":"ContainerDied","Data":"9226b5319566a1063de47071898ce2947388e43c185531382cb8a9b7afebebbc"} Oct 30 18:19:30.292802 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:30.292786 4603 scope.go:122] "RemoveContainer" containerID="6d94ee3965a0d533126fcfa61d335409ccf573b4ef5822e10a1395fa3a47deac" Oct 30 18:19:30.293099 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:19:30.293079 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d94ee3965a0d533126fcfa61d335409ccf573b4ef5822e10a1395fa3a47deac\": container with ID starting with 6d94ee3965a0d533126fcfa61d335409ccf573b4ef5822e10a1395fa3a47deac not found: ID does not exist" containerID="6d94ee3965a0d533126fcfa61d335409ccf573b4ef5822e10a1395fa3a47deac" Oct 30 18:19:30.293182 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:30.293105 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d94ee3965a0d533126fcfa61d335409ccf573b4ef5822e10a1395fa3a47deac"} err="failed to get container status \"6d94ee3965a0d533126fcfa61d335409ccf573b4ef5822e10a1395fa3a47deac\": rpc error: code = NotFound desc = could not find container \"6d94ee3965a0d533126fcfa61d335409ccf573b4ef5822e10a1395fa3a47deac\": container with ID starting with 6d94ee3965a0d533126fcfa61d335409ccf573b4ef5822e10a1395fa3a47deac not found: ID does not exist" Oct 30 18:19:30.293182 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:30.293122 4603 scope.go:122] "RemoveContainer" containerID="a27e5ec44600057d403fb860e9675144e27c797945be7f4cb1b40c78d5ec00b0" Oct 30 18:19:30.312297 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:30.312269 4603 scope.go:122] "RemoveContainer" containerID="cd9faa3b3a83d54fc93fc5f072721bd4c1d303273fc7835188352f5b849a5827" Oct 30 18:19:30.321929 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:30.321913 4603 scope.go:122] "RemoveContainer" containerID="a27e5ec44600057d403fb860e9675144e27c797945be7f4cb1b40c78d5ec00b0" Oct 30 18:19:30.322293 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:19:30.322271 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a27e5ec44600057d403fb860e9675144e27c797945be7f4cb1b40c78d5ec00b0\": container with ID starting with a27e5ec44600057d403fb860e9675144e27c797945be7f4cb1b40c78d5ec00b0 not found: ID does not exist" containerID="a27e5ec44600057d403fb860e9675144e27c797945be7f4cb1b40c78d5ec00b0" Oct 30 18:19:30.322359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:30.322305 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27e5ec44600057d403fb860e9675144e27c797945be7f4cb1b40c78d5ec00b0"} err="failed to get container status \"a27e5ec44600057d403fb860e9675144e27c797945be7f4cb1b40c78d5ec00b0\": rpc error: code = NotFound desc = could not find container \"a27e5ec44600057d403fb860e9675144e27c797945be7f4cb1b40c78d5ec00b0\": container with ID starting with a27e5ec44600057d403fb860e9675144e27c797945be7f4cb1b40c78d5ec00b0 not found: ID does not exist" Oct 30 18:19:30.322359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:30.322330 4603 scope.go:122] "RemoveContainer" containerID="cd9faa3b3a83d54fc93fc5f072721bd4c1d303273fc7835188352f5b849a5827" Oct 30 18:19:30.322656 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:19:30.322638 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd9faa3b3a83d54fc93fc5f072721bd4c1d303273fc7835188352f5b849a5827\": container with ID starting with cd9faa3b3a83d54fc93fc5f072721bd4c1d303273fc7835188352f5b849a5827 not found: ID does not exist" containerID="cd9faa3b3a83d54fc93fc5f072721bd4c1d303273fc7835188352f5b849a5827" Oct 30 18:19:30.322737 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:30.322662 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9faa3b3a83d54fc93fc5f072721bd4c1d303273fc7835188352f5b849a5827"} err="failed to get container status \"cd9faa3b3a83d54fc93fc5f072721bd4c1d303273fc7835188352f5b849a5827\": rpc error: code = NotFound desc = could not find container \"cd9faa3b3a83d54fc93fc5f072721bd4c1d303273fc7835188352f5b849a5827\": container with ID starting with cd9faa3b3a83d54fc93fc5f072721bd4c1d303273fc7835188352f5b849a5827 not found: ID does not exist" Oct 30 18:19:30.335401 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:30.335378 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w"] Oct 30 18:19:30.346933 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:30.346914 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-zc68w"] Oct 30 18:19:30.371490 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:30.371469 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq"] Oct 30 18:19:30.377419 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:30.377402 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-khqcq"] Oct 30 18:19:31.108871 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.108815 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:31.256483 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.256451 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:31.325673 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.325646 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d"] Oct 30 18:19:31.326274 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.326260 4603 cpu_manager.go:414] "containerMap: removing container" podUID="0ce9abc7-557e-4e1b-885c-410e0f51164c" containerName="istio-proxy" Oct 30 18:19:31.326324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.326274 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ce9abc7-557e-4e1b-885c-410e0f51164c" containerName="istio-proxy" Oct 30 18:19:31.326324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.326285 4603 cpu_manager.go:414] "containerMap: removing container" podUID="32a762e7-af53-435f-b602-f7ef85870b8e" containerName="payload-processing" Oct 30 18:19:31.326324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.326291 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32a762e7-af53-435f-b602-f7ef85870b8e" containerName="payload-processing" Oct 30 18:19:31.326324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.326310 4603 cpu_manager.go:414] "containerMap: removing container" podUID="f048e3e7-0037-4d5f-93ae-eedf127e62a5" containerName="payload-pre-processing" Oct 30 18:19:31.326324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.326315 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f048e3e7-0037-4d5f-93ae-eedf127e62a5" containerName="payload-pre-processing" Oct 30 18:19:31.326324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.326325 4603 cpu_manager.go:414] "containerMap: removing container" podUID="02a7d3cd-7df0-41c6-9063-f2caf0e934d4" containerName="istio-proxy" Oct 30 18:19:31.326538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.326331 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02a7d3cd-7df0-41c6-9063-f2caf0e934d4" containerName="istio-proxy" Oct 30 18:19:31.326538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.326350 4603 cpu_manager.go:414] "containerMap: removing container" podUID="02a7d3cd-7df0-41c6-9063-f2caf0e934d4" containerName="istio-proxy" Oct 30 18:19:31.326538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.326355 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02a7d3cd-7df0-41c6-9063-f2caf0e934d4" containerName="istio-proxy" Oct 30 18:19:31.326538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.326453 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="32a762e7-af53-435f-b602-f7ef85870b8e" containerName="payload-processing" Oct 30 18:19:31.326538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.326464 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="f048e3e7-0037-4d5f-93ae-eedf127e62a5" containerName="payload-pre-processing" Oct 30 18:19:31.326538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.326472 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="02a7d3cd-7df0-41c6-9063-f2caf0e934d4" containerName="istio-proxy" Oct 30 18:19:31.326538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.326480 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="02a7d3cd-7df0-41c6-9063-f2caf0e934d4" containerName="istio-proxy" Oct 30 18:19:31.326538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.326490 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ce9abc7-557e-4e1b-885c-410e0f51164c" containerName="istio-proxy" Oct 30 18:19:31.331354 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.331339 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d" Oct 30 18:19:31.365665 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.365604 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d"] Oct 30 18:19:31.376209 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.376186 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a3be60\"" Oct 30 18:19:31.376407 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.376343 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a3be60-dockercfg-9658c\"" Oct 30 18:19:31.395000 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.394977 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f77e84fd-cfe3-47be-9a3a-58759d5a9fc8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d\" (UID: \"f77e84fd-cfe3-47be-9a3a-58759d5a9fc8\") " pod="openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d" Oct 30 18:19:31.395098 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.395033 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb7kf\" (UniqueName: \"kubernetes.io/projected/f77e84fd-cfe3-47be-9a3a-58759d5a9fc8-kube-api-access-sb7kf\") pod \"payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d\" (UID: \"f77e84fd-cfe3-47be-9a3a-58759d5a9fc8\") " pod="openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d" Oct 30 18:19:31.412441 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.412416 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02a7d3cd-7df0-41c6-9063-f2caf0e934d4" path="/var/lib/kubelet/pods/02a7d3cd-7df0-41c6-9063-f2caf0e934d4/volumes" Oct 30 18:19:31.412966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.412952 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ce9abc7-557e-4e1b-885c-410e0f51164c" path="/var/lib/kubelet/pods/0ce9abc7-557e-4e1b-885c-410e0f51164c/volumes" Oct 30 18:19:31.500596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.497673 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb7kf\" (UniqueName: \"kubernetes.io/projected/f77e84fd-cfe3-47be-9a3a-58759d5a9fc8-kube-api-access-sb7kf\") pod \"payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d\" (UID: \"f77e84fd-cfe3-47be-9a3a-58759d5a9fc8\") " pod="openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d" Oct 30 18:19:31.505135 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.505081 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f77e84fd-cfe3-47be-9a3a-58759d5a9fc8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d\" (UID: \"f77e84fd-cfe3-47be-9a3a-58759d5a9fc8\") " pod="openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d" Oct 30 18:19:31.532650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.525646 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f77e84fd-cfe3-47be-9a3a-58759d5a9fc8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d\" (UID: \"f77e84fd-cfe3-47be-9a3a-58759d5a9fc8\") " pod="openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d" Oct 30 18:19:31.532900 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.532426 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:31.632572 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.630314 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb7kf\" (UniqueName: \"kubernetes.io/projected/f77e84fd-cfe3-47be-9a3a-58759d5a9fc8-kube-api-access-sb7kf\") pod \"payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d\" (UID: \"f77e84fd-cfe3-47be-9a3a-58759d5a9fc8\") " pod="openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d" Oct 30 18:19:31.648415 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.647558 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d" Oct 30 18:19:31.722091 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.716364 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:31.858493 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:31.858456 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:32.137251 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:32.137188 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d"] Oct 30 18:19:32.138145 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:19:32.138117 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf77e84fd_cfe3_47be_9a3a_58759d5a9fc8.slice/crio-6810da7dbe7592b0a30e24c63976a3f26953d2f5713bd8acdf38709f455c5168 WatchSource:0}: Error finding container 6810da7dbe7592b0a30e24c63976a3f26953d2f5713bd8acdf38709f455c5168: Status 404 returned error can't find the container with id 6810da7dbe7592b0a30e24c63976a3f26953d2f5713bd8acdf38709f455c5168 Oct 30 18:19:32.302276 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:32.302247 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d" event={"ID":"f77e84fd-cfe3-47be-9a3a-58759d5a9fc8","Type":"ContainerStarted","Data":"b7c3e014b353eb0e0602ec884ea5a3322efc232b775ba721b49cd6c79f32da3d"} Oct 30 18:19:32.302391 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:32.302299 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d" event={"ID":"f77e84fd-cfe3-47be-9a3a-58759d5a9fc8","Type":"ContainerStarted","Data":"6810da7dbe7592b0a30e24c63976a3f26953d2f5713bd8acdf38709f455c5168"} Oct 30 18:19:32.302435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:32.302401 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d" Oct 30 18:19:32.346897 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:32.346852 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d" podStartSLOduration=1.346838664 podStartE2EDuration="1.346838664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:32.344416607 +0000 UTC m=+1633.059986482" watchObservedRunningTime="2026-10-30 18:19:32.346838664 +0000 UTC m=+1633.062408564" Oct 30 18:19:33.264715 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:33.264684 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:34.717463 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:34.717425 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:34.880512 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:34.880460 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:35.595882 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:35.595846 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:35.735064 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:35.735026 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:35.873324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:35.873246 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:36.792996 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:36.792966 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:36.935423 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:36.935391 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:36.952979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:36.952953 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn"] Oct 30 18:19:36.958218 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:36.958202 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn" Oct 30 18:19:36.985798 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:36.985769 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-069812\"" Oct 30 18:19:36.985964 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:36.985780 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-069812-dockercfg-4hd4x\"" Oct 30 18:19:36.991744 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:36.991720 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn"] Oct 30 18:19:36.995085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:36.995062 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqz24\" (UniqueName: \"kubernetes.io/projected/2846f1c3-f62a-4e62-b334-d0fa80036f4a-kube-api-access-nqz24\") pod \"payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn\" (UID: \"2846f1c3-f62a-4e62-b334-d0fa80036f4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn" Oct 30 18:19:36.995264 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:36.995250 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2846f1c3-f62a-4e62-b334-d0fa80036f4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn\" (UID: \"2846f1c3-f62a-4e62-b334-d0fa80036f4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn" Oct 30 18:19:37.177975 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:37.177886 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqz24\" (UniqueName: \"kubernetes.io/projected/2846f1c3-f62a-4e62-b334-d0fa80036f4a-kube-api-access-nqz24\") pod \"payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn\" (UID: \"2846f1c3-f62a-4e62-b334-d0fa80036f4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn" Oct 30 18:19:37.178854 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:37.178825 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2846f1c3-f62a-4e62-b334-d0fa80036f4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn\" (UID: \"2846f1c3-f62a-4e62-b334-d0fa80036f4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn" Oct 30 18:19:37.179895 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:37.179726 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2846f1c3-f62a-4e62-b334-d0fa80036f4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn\" (UID: \"2846f1c3-f62a-4e62-b334-d0fa80036f4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn" Oct 30 18:19:37.284045 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:37.280688 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g"] Oct 30 18:19:37.345541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:37.306209 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g" Oct 30 18:19:37.345541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:37.308388 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g"] Oct 30 18:19:37.345541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:37.314532 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqz24\" (UniqueName: \"kubernetes.io/projected/2846f1c3-f62a-4e62-b334-d0fa80036f4a-kube-api-access-nqz24\") pod \"payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn\" (UID: \"2846f1c3-f62a-4e62-b334-d0fa80036f4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn" Oct 30 18:19:37.574989 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:37.574836 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn" Oct 30 18:19:37.582920 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:37.576861 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsqrt\" (UniqueName: \"kubernetes.io/projected/9bd393cb-cdd9-4d66-ac02-ee4e78f0062f-kube-api-access-gsqrt\") pod \"payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g\" (UID: \"9bd393cb-cdd9-4d66-ac02-ee4e78f0062f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g" Oct 30 18:19:37.588018 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:37.583481 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bd393cb-cdd9-4d66-ac02-ee4e78f0062f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g\" (UID: \"9bd393cb-cdd9-4d66-ac02-ee4e78f0062f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g" Oct 30 18:19:37.668020 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:37.667979 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:37.707570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:37.705815 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsqrt\" (UniqueName: \"kubernetes.io/projected/9bd393cb-cdd9-4d66-ac02-ee4e78f0062f-kube-api-access-gsqrt\") pod \"payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g\" (UID: \"9bd393cb-cdd9-4d66-ac02-ee4e78f0062f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g" Oct 30 18:19:37.707570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:37.706121 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bd393cb-cdd9-4d66-ac02-ee4e78f0062f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g\" (UID: \"9bd393cb-cdd9-4d66-ac02-ee4e78f0062f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g" Oct 30 18:19:37.718381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:37.717615 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bd393cb-cdd9-4d66-ac02-ee4e78f0062f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g\" (UID: \"9bd393cb-cdd9-4d66-ac02-ee4e78f0062f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g" Oct 30 18:19:37.774527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:37.767272 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsqrt\" (UniqueName: \"kubernetes.io/projected/9bd393cb-cdd9-4d66-ac02-ee4e78f0062f-kube-api-access-gsqrt\") pod \"payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g\" (UID: \"9bd393cb-cdd9-4d66-ac02-ee4e78f0062f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g" Oct 30 18:19:38.040379 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:38.040353 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn"] Oct 30 18:19:38.041953 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:19:38.041925 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2846f1c3_f62a_4e62_b334_d0fa80036f4a.slice/crio-7998e599000c874ec5afb4d067188ffcc13f90e38470dd6c21a60811d4af47cb WatchSource:0}: Error finding container 7998e599000c874ec5afb4d067188ffcc13f90e38470dd6c21a60811d4af47cb: Status 404 returned error can't find the container with id 7998e599000c874ec5afb4d067188ffcc13f90e38470dd6c21a60811d4af47cb Oct 30 18:19:38.056981 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:38.056952 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g" Oct 30 18:19:38.371076 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:19:38.371047 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd393cb_cdd9_4d66_ac02_ee4e78f0062f.slice/crio-d2db19db65847039adcaf036780d73566702bc3e29e6816ff4f1a52f72ba7e44 WatchSource:0}: Error finding container d2db19db65847039adcaf036780d73566702bc3e29e6816ff4f1a52f72ba7e44: Status 404 returned error can't find the container with id d2db19db65847039adcaf036780d73566702bc3e29e6816ff4f1a52f72ba7e44 Oct 30 18:19:38.371186 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:38.371148 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g"] Oct 30 18:19:38.457285 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:38.457252 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g" event={"ID":"9bd393cb-cdd9-4d66-ac02-ee4e78f0062f","Type":"ContainerStarted","Data":"ecce2244bd5a2ddec6586c572836bf384490d940955559f4eeff56583c11b917"} Oct 30 18:19:38.457435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:38.457320 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g" event={"ID":"9bd393cb-cdd9-4d66-ac02-ee4e78f0062f","Type":"ContainerStarted","Data":"d2db19db65847039adcaf036780d73566702bc3e29e6816ff4f1a52f72ba7e44"} Oct 30 18:19:38.457435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:38.457359 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g" Oct 30 18:19:38.458966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:38.458935 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn" event={"ID":"2846f1c3-f62a-4e62-b334-d0fa80036f4a","Type":"ContainerStarted","Data":"8bc9cc8979470164b7a5365c499015e038c38dae7a978c591edd5c49e6a123a9"} Oct 30 18:19:38.459067 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:38.458994 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn" event={"ID":"2846f1c3-f62a-4e62-b334-d0fa80036f4a","Type":"ContainerStarted","Data":"7998e599000c874ec5afb4d067188ffcc13f90e38470dd6c21a60811d4af47cb"} Oct 30 18:19:38.459067 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:38.459026 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn" Oct 30 18:19:38.498752 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:38.498715 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g" podStartSLOduration=1.498703619 podStartE2EDuration="1.498703619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:38.496840176 +0000 UTC m=+1639.212410050" watchObservedRunningTime="2026-10-30 18:19:38.498703619 +0000 UTC m=+1639.214273492" Oct 30 18:19:38.539574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:38.539531 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn" podStartSLOduration=2.539516768 podStartE2EDuration="2.539516768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:38.535786206 +0000 UTC m=+1639.251356082" watchObservedRunningTime="2026-10-30 18:19:38.539516768 +0000 UTC m=+1639.255086636" Oct 30 18:19:39.377583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:39.377542 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:41.020289 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:41.020257 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:42.176865 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:42.176835 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:42.459827 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:42.449301 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:42.606473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:42.593175 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt"] Oct 30 18:19:42.606473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:42.603101 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt" Oct 30 18:19:42.631074 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:42.631043 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d951c4\"" Oct 30 18:19:42.631392 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:42.631370 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d951c4-dockercfg-xnvfh\"" Oct 30 18:19:42.721575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:42.718570 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt"] Oct 30 18:19:42.757580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:42.751222 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74762a1d-083f-43ed-abb4-8ea80c4ba83c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt\" (UID: \"74762a1d-083f-43ed-abb4-8ea80c4ba83c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt" Oct 30 18:19:42.757580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:42.751438 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s99lx\" (UniqueName: \"kubernetes.io/projected/74762a1d-083f-43ed-abb4-8ea80c4ba83c-kube-api-access-s99lx\") pod \"payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt\" (UID: \"74762a1d-083f-43ed-abb4-8ea80c4ba83c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt" Oct 30 18:19:42.853589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:42.853557 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s99lx\" (UniqueName: \"kubernetes.io/projected/74762a1d-083f-43ed-abb4-8ea80c4ba83c-kube-api-access-s99lx\") pod \"payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt\" (UID: \"74762a1d-083f-43ed-abb4-8ea80c4ba83c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt" Oct 30 18:19:42.854058 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:42.854019 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74762a1d-083f-43ed-abb4-8ea80c4ba83c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt\" (UID: \"74762a1d-083f-43ed-abb4-8ea80c4ba83c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt" Oct 30 18:19:42.855220 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:42.855196 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74762a1d-083f-43ed-abb4-8ea80c4ba83c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt\" (UID: \"74762a1d-083f-43ed-abb4-8ea80c4ba83c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt" Oct 30 18:19:42.887287 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:42.884897 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv"] Oct 30 18:19:42.888760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:42.888734 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s99lx\" (UniqueName: \"kubernetes.io/projected/74762a1d-083f-43ed-abb4-8ea80c4ba83c-kube-api-access-s99lx\") pod \"payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt\" (UID: \"74762a1d-083f-43ed-abb4-8ea80c4ba83c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt" Oct 30 18:19:42.890706 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:42.890683 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv"] Oct 30 18:19:42.890827 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:42.890781 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv" Oct 30 18:19:43.040763 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:43.040735 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt" Oct 30 18:19:43.057419 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:43.057392 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4n5n\" (UniqueName: \"kubernetes.io/projected/0f4b207b-5280-4361-9e36-a64feaf49cff-kube-api-access-v4n5n\") pod \"payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv\" (UID: \"0f4b207b-5280-4361-9e36-a64feaf49cff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv" Oct 30 18:19:43.057651 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:43.057633 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4b207b-5280-4361-9e36-a64feaf49cff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv\" (UID: \"0f4b207b-5280-4361-9e36-a64feaf49cff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv" Oct 30 18:19:43.159766 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:43.159732 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4n5n\" (UniqueName: \"kubernetes.io/projected/0f4b207b-5280-4361-9e36-a64feaf49cff-kube-api-access-v4n5n\") pod \"payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv\" (UID: \"0f4b207b-5280-4361-9e36-a64feaf49cff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv" Oct 30 18:19:43.160021 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:43.160002 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4b207b-5280-4361-9e36-a64feaf49cff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv\" (UID: \"0f4b207b-5280-4361-9e36-a64feaf49cff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv" Oct 30 18:19:43.160802 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:43.160774 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4b207b-5280-4361-9e36-a64feaf49cff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv\" (UID: \"0f4b207b-5280-4361-9e36-a64feaf49cff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv" Oct 30 18:19:43.183233 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:43.183203 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4n5n\" (UniqueName: \"kubernetes.io/projected/0f4b207b-5280-4361-9e36-a64feaf49cff-kube-api-access-v4n5n\") pod \"payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv\" (UID: \"0f4b207b-5280-4361-9e36-a64feaf49cff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv" Oct 30 18:19:43.220901 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:43.220874 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv" Oct 30 18:19:43.241719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:43.241687 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:43.352740 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:43.352649 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d" Oct 30 18:19:43.394848 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:43.394127 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt"] Oct 30 18:19:43.637394 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:43.637298 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt" event={"ID":"74762a1d-083f-43ed-abb4-8ea80c4ba83c","Type":"ContainerStarted","Data":"99f31cc41833897c405c9af8d272a3c45ed05a2a5e376727a4d6efc6891a3c2d"} Oct 30 18:19:43.637394 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:43.637364 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt" event={"ID":"74762a1d-083f-43ed-abb4-8ea80c4ba83c","Type":"ContainerStarted","Data":"ac249cf57eaea495bc9eb10decb15bd0ba502c97bf16c45d538ad472297cc4f0"} Oct 30 18:19:43.637394 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:43.637392 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt" Oct 30 18:19:43.653520 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:19:43.653475 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4b207b_5280_4361_9e36_a64feaf49cff.slice/crio-64fddb07734f8041f16456ec15006aa304437f5a837b1d13d70891b5c1136168 WatchSource:0}: Error finding container 64fddb07734f8041f16456ec15006aa304437f5a837b1d13d70891b5c1136168: Status 404 returned error can't find the container with id 64fddb07734f8041f16456ec15006aa304437f5a837b1d13d70891b5c1136168 Oct 30 18:19:43.653520 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:43.653488 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv"] Oct 30 18:19:43.679912 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:43.679871 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt" podStartSLOduration=1.6798570499999999 podStartE2EDuration="1.67985705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:43.676890465 +0000 UTC m=+1644.392460340" watchObservedRunningTime="2026-10-30 18:19:43.67985705 +0000 UTC m=+1644.395426966" Oct 30 18:19:44.645216 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:44.645178 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv" event={"ID":"0f4b207b-5280-4361-9e36-a64feaf49cff","Type":"ContainerStarted","Data":"082a296c35177610e0222286fd6d9c789ef0a19ea73b70bedb29469dbbe1e880"} Oct 30 18:19:44.645641 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:44.645242 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv" event={"ID":"0f4b207b-5280-4361-9e36-a64feaf49cff","Type":"ContainerStarted","Data":"64fddb07734f8041f16456ec15006aa304437f5a837b1d13d70891b5c1136168"} Oct 30 18:19:44.645641 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:44.645275 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv" Oct 30 18:19:44.687271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:44.687225 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv" podStartSLOduration=2.6872096450000003 podStartE2EDuration="2.687209645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:44.685019633 +0000 UTC m=+1645.400589529" watchObservedRunningTime="2026-10-30 18:19:44.687209645 +0000 UTC m=+1645.402779519" Oct 30 18:19:44.868725 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:44.868693 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:47.098531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:47.090890 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:48.634349 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:48.634310 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:49.497639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:49.497606 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn" Oct 30 18:19:49.498004 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:49.497987 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g" Oct 30 18:19:50.336223 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:50.336186 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:51.931972 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:19:51.931935 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:19:52.527383 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:52.527347 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:52.680567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:52.680532 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:54.346417 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:54.346378 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:54.646946 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:54.646871 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt" Oct 30 18:19:55.655093 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:55.655063 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv" Oct 30 18:19:56.046213 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:56.046179 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:57.817532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:57.817479 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:58.490084 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:58.490052 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:58.968219 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:58.968186 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:59.116456 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:19:59.116425 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:00.089359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:00.089315 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:00.245901 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:00.245860 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:00.426265 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:00.421198 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:01.215966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:01.215924 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:01.489577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:01.488934 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:02.114556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:02.113967 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:02.773545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:02.773517 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:02.939149 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:02.939118 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:04.054963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:04.054931 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:05.171274 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:05.171239 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:05.315698 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:05.315660 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:05.318486 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:05.318459 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg"] Oct 30 18:20:05.327758 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:05.327737 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg" Oct 30 18:20:05.341938 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:05.341915 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg"] Oct 30 18:20:05.349272 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:05.349244 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-394a31\"" Oct 30 18:20:05.349428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:05.349263 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-394a31-dockercfg-hvtsc\"" Oct 30 18:20:05.517047 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:05.498778 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plfnt\" (UniqueName: \"kubernetes.io/projected/ea551dc1-ef6e-4a74-a118-bef1b0f70a57-kube-api-access-plfnt\") pod \"payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg\" (UID: \"ea551dc1-ef6e-4a74-a118-bef1b0f70a57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg" Oct 30 18:20:05.519464 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:05.519434 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea551dc1-ef6e-4a74-a118-bef1b0f70a57-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg\" (UID: \"ea551dc1-ef6e-4a74-a118-bef1b0f70a57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg" Oct 30 18:20:05.621905 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:05.621455 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plfnt\" (UniqueName: \"kubernetes.io/projected/ea551dc1-ef6e-4a74-a118-bef1b0f70a57-kube-api-access-plfnt\") pod \"payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg\" (UID: \"ea551dc1-ef6e-4a74-a118-bef1b0f70a57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg" Oct 30 18:20:05.622269 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:05.622241 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea551dc1-ef6e-4a74-a118-bef1b0f70a57-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg\" (UID: \"ea551dc1-ef6e-4a74-a118-bef1b0f70a57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg" Oct 30 18:20:05.633174 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:05.633144 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea551dc1-ef6e-4a74-a118-bef1b0f70a57-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg\" (UID: \"ea551dc1-ef6e-4a74-a118-bef1b0f70a57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg" Oct 30 18:20:05.714168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:05.714135 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf"] Oct 30 18:20:05.719998 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:05.719953 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf"] Oct 30 18:20:05.731471 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:05.726036 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf" Oct 30 18:20:05.745525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:05.745395 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plfnt\" (UniqueName: \"kubernetes.io/projected/ea551dc1-ef6e-4a74-a118-bef1b0f70a57-kube-api-access-plfnt\") pod \"payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg\" (UID: \"ea551dc1-ef6e-4a74-a118-bef1b0f70a57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg" Oct 30 18:20:05.827178 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:05.827151 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa3594b-6dab-4f71-a8c1-183151163cfb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf\" (UID: \"6fa3594b-6dab-4f71-a8c1-183151163cfb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf" Oct 30 18:20:05.827378 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:05.827354 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45x2d\" (UniqueName: \"kubernetes.io/projected/6fa3594b-6dab-4f71-a8c1-183151163cfb-kube-api-access-45x2d\") pod \"payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf\" (UID: \"6fa3594b-6dab-4f71-a8c1-183151163cfb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf" Oct 30 18:20:05.928303 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:05.928275 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa3594b-6dab-4f71-a8c1-183151163cfb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf\" (UID: \"6fa3594b-6dab-4f71-a8c1-183151163cfb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf" Oct 30 18:20:05.928473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:05.928452 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45x2d\" (UniqueName: \"kubernetes.io/projected/6fa3594b-6dab-4f71-a8c1-183151163cfb-kube-api-access-45x2d\") pod \"payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf\" (UID: \"6fa3594b-6dab-4f71-a8c1-183151163cfb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf" Oct 30 18:20:05.929061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:05.929037 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa3594b-6dab-4f71-a8c1-183151163cfb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf\" (UID: \"6fa3594b-6dab-4f71-a8c1-183151163cfb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf" Oct 30 18:20:05.946102 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:05.946083 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg" Oct 30 18:20:05.951710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:05.951688 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45x2d\" (UniqueName: \"kubernetes.io/projected/6fa3594b-6dab-4f71-a8c1-183151163cfb-kube-api-access-45x2d\") pod \"payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf\" (UID: \"6fa3594b-6dab-4f71-a8c1-183151163cfb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf" Oct 30 18:20:06.082883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:06.082800 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf" Oct 30 18:20:06.143464 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:06.143367 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:06.318981 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:06.318945 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg"] Oct 30 18:20:06.335247 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:20:06.335059 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea551dc1_ef6e_4a74_a118_bef1b0f70a57.slice/crio-93004c52b72962190bc9b20925c1bd4cd04814638cc78b5aad376fddce7e996c WatchSource:0}: Error finding container 93004c52b72962190bc9b20925c1bd4cd04814638cc78b5aad376fddce7e996c: Status 404 returned error can't find the container with id 93004c52b72962190bc9b20925c1bd4cd04814638cc78b5aad376fddce7e996c Oct 30 18:20:06.393124 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:20:06.393100 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa3594b_6dab_4f71_a8c1_183151163cfb.slice/crio-d894bf625f6bb63c5c1a863563e71d6a791e5f0f0813513e58101590d9aa7e18 WatchSource:0}: Error finding container d894bf625f6bb63c5c1a863563e71d6a791e5f0f0813513e58101590d9aa7e18: Status 404 returned error can't find the container with id d894bf625f6bb63c5c1a863563e71d6a791e5f0f0813513e58101590d9aa7e18 Oct 30 18:20:06.394301 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:06.394276 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf"] Oct 30 18:20:06.986463 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:06.986432 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf" event={"ID":"6fa3594b-6dab-4f71-a8c1-183151163cfb","Type":"ContainerStarted","Data":"c08244d35a17e2933f026ea8da610ca34689a82f4cfb56e1b1a3c1c802209422"} Oct 30 18:20:06.986662 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:06.986486 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf" event={"ID":"6fa3594b-6dab-4f71-a8c1-183151163cfb","Type":"ContainerStarted","Data":"d894bf625f6bb63c5c1a863563e71d6a791e5f0f0813513e58101590d9aa7e18"} Oct 30 18:20:06.986662 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:06.986529 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf" Oct 30 18:20:06.988187 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:06.988163 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg" event={"ID":"ea551dc1-ef6e-4a74-a118-bef1b0f70a57","Type":"ContainerStarted","Data":"f9d01f34e96b92680ab80d535949fec53341e396e02ba2a0050f00c7f086d1f2"} Oct 30 18:20:06.988329 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:06.988212 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg" Oct 30 18:20:06.988329 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:06.988237 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg" event={"ID":"ea551dc1-ef6e-4a74-a118-bef1b0f70a57","Type":"ContainerStarted","Data":"93004c52b72962190bc9b20925c1bd4cd04814638cc78b5aad376fddce7e996c"} Oct 30 18:20:07.027456 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:07.027285 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf" podStartSLOduration=2.02726859 podStartE2EDuration="2.02726859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:07.025005228 +0000 UTC m=+1667.740575114" watchObservedRunningTime="2026-10-30 18:20:07.02726859 +0000 UTC m=+1667.742838466" Oct 30 18:20:07.064833 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:07.064788 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg" podStartSLOduration=2.064777085 podStartE2EDuration="2.064777085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:07.064316241 +0000 UTC m=+1667.779886115" watchObservedRunningTime="2026-10-30 18:20:07.064777085 +0000 UTC m=+1667.780346960" Oct 30 18:20:07.941298 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:07.941251 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:09.261272 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:09.261234 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:09.426770 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:09.426310 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:09.588859 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:09.587113 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:10.104524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:10.098840 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:11.383926 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:11.383891 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:11.771005 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:11.770975 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:13.939008 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:13.938972 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:15.037357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:15.037325 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:20:15.598907 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:15.598873 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:15.766778 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:15.766744 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:17.477121 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:17.477087 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:18.096709 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:18.096677 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg" Oct 30 18:20:18.104695 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:18.104668 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf" Oct 30 18:20:18.658694 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:18.652641 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Oct 30 18:20:18.735532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:18.734582 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:18.935914 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:18.935803 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:18.980653 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:18.980587 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:20:19.075640 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:19.075600 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:19.080609 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:19.080573 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:19.648762 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:19.648703 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:19.864531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:19.843030 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:20.036532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:20.028786 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:20:20.062530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:20.028868 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:20:20.062530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:20.053198 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:20:20.089535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:20.080818 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:20.089535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:20.081301 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:20:20.106533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:20.101821 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:20.106533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:20.102781 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:20:20.106533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:20.103452 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:20:20.106533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:20.103541 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:20:20.132964 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:20.132479 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:20.451578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:20.422793 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Oct 30 18:20:20.732580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:20.660286 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:20.759677 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:20.759640 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:20:20.829753 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:20.829713 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:20:21.004699 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:21.004573 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:21.060895 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:21.060850 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:21.090551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:21.089836 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:21.130057 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:21.129767 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:20:21.149816 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:21.149783 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:21.159612 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:21.153978 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:21.174730 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:21.174699 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:20:21.739961 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:21.739917 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:21.835889 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:21.835833 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:21.836678 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:21.836653 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:20:22.008594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:22.005766 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:22.098532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:22.076245 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:22.121613 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:22.121565 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:20:22.161763 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:22.143916 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:22.182294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:22.182257 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:22.236051 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:22.182551 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:20:22.574365 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:20:22.573685 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:20:22.742749 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:22.742281 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:22.842685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:22.842583 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:20:22.977184 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:22.977138 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:23.050196 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:23.050150 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:23.091529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:23.083176 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:23.170582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:23.137090 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:20:23.170582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:23.165662 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:23.252531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:23.249167 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:23.264575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:23.260669 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:20:23.776530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:23.753629 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:24.044587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:23.970064 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:24.094640 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:24.094573 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:24.193576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:24.189826 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:20:24.303643 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:24.197799 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:24.333144 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:24.283957 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:24.334968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:24.311581 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:20:24.393680 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:24.393628 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:24.739586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:24.715018 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:24.763837 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:24.763790 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:25.044724 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:25.044687 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:20:25.131532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:25.117646 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:25.213532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:25.209638 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:20:25.213532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:25.209726 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" podUID="e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:25.301693 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:25.301590 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:25.341073 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:25.341028 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:25.341741 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:25.341327 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:20:25.342740 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:25.341356 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:25.404746 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:25.404298 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:25.512321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:25.510012 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:25.764743 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:25.764595 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:20:26.111532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:26.110542 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:26.122320 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:26.122271 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:26.302720 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:26.302660 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:26.358728 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:26.358670 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:20:26.359468 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:26.359423 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:20:26.360268 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:26.360246 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:26.416742 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:26.416637 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:26.788590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:26.776881 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:27.112677 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:27.112579 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:27.168625 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:27.168578 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:27.169201 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:27.169165 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:27.547622 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:27.499083 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:27.625455 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:27.563177 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:20:27.625455 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:27.563593 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:27.625455 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:27.563634 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:27.625455 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:27.567472 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:27.625455 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:27.567529 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:27.785690 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:27.785633 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:20:27.975617 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:27.954933 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:28.132415 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:28.132070 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:28.132914 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:28.132866 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:28.195534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:28.170668 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:28.567710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:28.563959 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 30 18:20:28.633527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:28.625593 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:20:28.633527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:28.626794 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Oct 30 18:20:28.633527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:28.626857 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:20:28.633527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:28.629840 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:20:28.657529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:28.641742 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:20:28.657529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:28.642572 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:20:28.790532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:28.787597 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:20:29.094531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:29.092783 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:20:29.094531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:29.092865 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" podUID="22e0b280-2b65-4fad-8344-67049190d108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:29.143529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:29.138614 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:20:29.200530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:29.199623 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:20:29.431714 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:29.431618 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:29.486523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:29.476973 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:20:29.486523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:29.477040 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" podUID="5411300d-c1aa-4397-be3d-0ce869a1c1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:29.635658 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:29.635611 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:20:29.636660 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:29.635808 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:29.636964 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:29.635884 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:20:29.660531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:29.645948 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:20:29.714529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:29.714400 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:29.803741 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:29.797717 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:20:29.900713 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:29.900679 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:20:29.946805 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:29.946772 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:20:30.138954 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:30.138909 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:20:30.324526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:30.319309 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Oct 30 18:20:30.433628 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:30.433464 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:20:30.473522 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:30.470999 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:20:30.521029 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:30.514570 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:30.533523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:30.530088 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:20:30.663635 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:30.663597 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:30.692858 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:30.692788 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:20:30.856898 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:30.856863 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:30.957833 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:30.938148 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:20:31.043584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:31.035112 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:20:31.134312 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:31.100268 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:20:31.217032 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:31.216947 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp"] Oct 30 18:20:31.448527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:31.341230 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:31.772488 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:31.772460 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp" Oct 30 18:20:31.787604 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:31.787577 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a8dc70-dockercfg-ln2qg\"" Oct 30 18:20:31.788392 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:31.788359 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a8dc70\"" Oct 30 18:20:31.876534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:31.875446 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67deb4fb-e2e9-41ea-b2e9-530f568d96ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp\" (UID: \"67deb4fb-e2e9-41ea-b2e9-530f568d96ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp" Oct 30 18:20:31.876534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:31.875688 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7vt6\" (UniqueName: \"kubernetes.io/projected/67deb4fb-e2e9-41ea-b2e9-530f568d96ed-kube-api-access-k7vt6\") pod \"payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp\" (UID: \"67deb4fb-e2e9-41ea-b2e9-530f568d96ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp" Oct 30 18:20:31.887798 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:31.887762 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:20:31.887946 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:31.887808 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp"] Oct 30 18:20:31.979571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:31.978736 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67deb4fb-e2e9-41ea-b2e9-530f568d96ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp\" (UID: \"67deb4fb-e2e9-41ea-b2e9-530f568d96ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp" Oct 30 18:20:31.979571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:31.978993 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7vt6\" (UniqueName: \"kubernetes.io/projected/67deb4fb-e2e9-41ea-b2e9-530f568d96ed-kube-api-access-k7vt6\") pod \"payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp\" (UID: \"67deb4fb-e2e9-41ea-b2e9-530f568d96ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp" Oct 30 18:20:31.980854 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:31.980824 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67deb4fb-e2e9-41ea-b2e9-530f568d96ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp\" (UID: \"67deb4fb-e2e9-41ea-b2e9-530f568d96ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp" Oct 30 18:20:31.982199 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:31.982174 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:32.262579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:32.257613 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7vt6\" (UniqueName: \"kubernetes.io/projected/67deb4fb-e2e9-41ea-b2e9-530f568d96ed-kube-api-access-k7vt6\") pod \"payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp\" (UID: \"67deb4fb-e2e9-41ea-b2e9-530f568d96ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp" Oct 30 18:20:32.487145 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:32.487121 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp" Oct 30 18:20:33.482222 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:33.482195 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp"] Oct 30 18:20:33.788350 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:33.788315 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp" event={"ID":"67deb4fb-e2e9-41ea-b2e9-530f568d96ed","Type":"ContainerStarted","Data":"38bd18fe8a7870ee2519aba62bed333f5ff80d8722d1e601d088ff1bff60d287"} Oct 30 18:20:33.788479 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:33.788371 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp" event={"ID":"67deb4fb-e2e9-41ea-b2e9-530f568d96ed","Type":"ContainerStarted","Data":"4b720ba95a4cf6ebcc9109e4ab6071bac991f447af3a58a6ed2215cc69cf4f50"} Oct 30 18:20:33.788479 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:33.788401 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp" Oct 30 18:20:33.833530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:33.831186 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp" podStartSLOduration=2.8311661949999998 podStartE2EDuration="2.831166195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:33.830096497 +0000 UTC m=+1694.545666372" watchObservedRunningTime="2026-10-30 18:20:33.831166195 +0000 UTC m=+1694.546736071" Oct 30 18:20:34.348177 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:34.348130 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:36.139455 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:36.137658 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:36.324738 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:36.306741 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:36.472063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:36.471976 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:38.167491 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:38.167452 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:39.419915 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:39.419879 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:39.622628 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:39.595234 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:39.961635 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:39.928764 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg"] Oct 30 18:20:40.129877 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.129842 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:20:40.130199 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.130152 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" podUID="904c45e3-164e-45ee-b877-b8b565fbb7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:40.149659 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.148680 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg" Oct 30 18:20:40.176313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.167744 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-189cda-dockercfg-p48js\"" Oct 30 18:20:40.176313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.167846 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-189cda\"" Oct 30 18:20:40.176313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.175087 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg"] Oct 30 18:20:40.176313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.175155 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf"] Oct 30 18:20:40.180843 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.180601 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf"] Oct 30 18:20:40.180843 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.180708 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf" Oct 30 18:20:40.251947 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.251908 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c302f2-7a93-46f4-9842-426a6622bfcd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf\" (UID: \"29c302f2-7a93-46f4-9842-426a6622bfcd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf" Oct 30 18:20:40.252127 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.251961 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2m6m\" (UniqueName: \"kubernetes.io/projected/9ad353f8-72ed-494e-ab3c-429d0169a45f-kube-api-access-k2m6m\") pod \"payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg\" (UID: \"9ad353f8-72ed-494e-ab3c-429d0169a45f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg" Oct 30 18:20:40.252127 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.252106 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad353f8-72ed-494e-ab3c-429d0169a45f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg\" (UID: \"9ad353f8-72ed-494e-ab3c-429d0169a45f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg" Oct 30 18:20:40.252325 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.252281 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7ch4\" (UniqueName: \"kubernetes.io/projected/29c302f2-7a93-46f4-9842-426a6622bfcd-kube-api-access-x7ch4\") pod \"payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf\" (UID: \"29c302f2-7a93-46f4-9842-426a6622bfcd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf" Oct 30 18:20:40.354968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.354897 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c302f2-7a93-46f4-9842-426a6622bfcd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf\" (UID: \"29c302f2-7a93-46f4-9842-426a6622bfcd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf" Oct 30 18:20:40.354968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.354956 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2m6m\" (UniqueName: \"kubernetes.io/projected/9ad353f8-72ed-494e-ab3c-429d0169a45f-kube-api-access-k2m6m\") pod \"payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg\" (UID: \"9ad353f8-72ed-494e-ab3c-429d0169a45f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg" Oct 30 18:20:40.355207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.355024 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad353f8-72ed-494e-ab3c-429d0169a45f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg\" (UID: \"9ad353f8-72ed-494e-ab3c-429d0169a45f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg" Oct 30 18:20:40.355207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.355192 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ch4\" (UniqueName: \"kubernetes.io/projected/29c302f2-7a93-46f4-9842-426a6622bfcd-kube-api-access-x7ch4\") pod \"payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf\" (UID: \"29c302f2-7a93-46f4-9842-426a6622bfcd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf" Oct 30 18:20:40.356199 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.355619 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c302f2-7a93-46f4-9842-426a6622bfcd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf\" (UID: \"29c302f2-7a93-46f4-9842-426a6622bfcd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf" Oct 30 18:20:40.356358 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.356333 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad353f8-72ed-494e-ab3c-429d0169a45f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg\" (UID: \"9ad353f8-72ed-494e-ab3c-429d0169a45f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg" Oct 30 18:20:40.379318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.379286 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7ch4\" (UniqueName: \"kubernetes.io/projected/29c302f2-7a93-46f4-9842-426a6622bfcd-kube-api-access-x7ch4\") pod \"payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf\" (UID: \"29c302f2-7a93-46f4-9842-426a6622bfcd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf" Oct 30 18:20:40.380114 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.380090 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2m6m\" (UniqueName: \"kubernetes.io/projected/9ad353f8-72ed-494e-ab3c-429d0169a45f-kube-api-access-k2m6m\") pod \"payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg\" (UID: \"9ad353f8-72ed-494e-ab3c-429d0169a45f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg" Oct 30 18:20:40.524308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.524271 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg" Oct 30 18:20:40.543585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.543388 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf" Oct 30 18:20:40.641017 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.640036 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:40.931558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.931420 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:40.986900 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:40.986866 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g"] Oct 30 18:20:41.036881 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:41.036836 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn"] Oct 30 18:20:41.138875 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:41.138754 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg"] Oct 30 18:20:41.159672 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:41.158914 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:20:41.159672 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:41.158986 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" podUID="30b23750-56cb-4895-80a8-8e54ae819f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:41.180533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:41.177662 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf"] Oct 30 18:20:41.210527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:41.210411 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:41.363674 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:41.363570 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:41.664215 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:41.664063 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:42.216428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:42.215154 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf" event={"ID":"29c302f2-7a93-46f4-9842-426a6622bfcd","Type":"ContainerStarted","Data":"7ad4e5ea9c987aea995059d605edc08d596bda2d328f79ecd3e7eacdfb65fdc5"} Oct 30 18:20:42.216428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:42.215217 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf" event={"ID":"29c302f2-7a93-46f4-9842-426a6622bfcd","Type":"ContainerStarted","Data":"982f59e9d8829c5b563bcdcf56488e3c3591f8331f1b8073a7572611f6c427c1"} Oct 30 18:20:42.216428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:42.216369 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf" Oct 30 18:20:42.239590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:42.239488 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg" event={"ID":"9ad353f8-72ed-494e-ab3c-429d0169a45f","Type":"ContainerStarted","Data":"7927b15a6433ed272d6308189bdd230b2b52ec78e1bb5e01707afcbadfe450b5"} Oct 30 18:20:42.239839 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:42.239821 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg" event={"ID":"9ad353f8-72ed-494e-ab3c-429d0169a45f","Type":"ContainerStarted","Data":"b0928025452fe6612c4715fc4ead80497baa2fd6dad6c90ce55d4b329c11758e"} Oct 30 18:20:42.240804 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:42.240780 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg" Oct 30 18:20:42.371195 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:42.371129 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf" podStartSLOduration=3.371109891 podStartE2EDuration="3.371109891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:42.368237576 +0000 UTC m=+1703.083807462" watchObservedRunningTime="2026-10-30 18:20:42.371109891 +0000 UTC m=+1703.086679765" Oct 30 18:20:42.422101 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:42.422067 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:42.453525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:42.436845 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg" podStartSLOduration=3.436825209 podStartE2EDuration="3.436825209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:42.427008792 +0000 UTC m=+1703.142578666" watchObservedRunningTime="2026-10-30 18:20:42.436825209 +0000 UTC m=+1703.152395086" Oct 30 18:20:42.765905 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:42.764624 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:42.916677 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:42.914645 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:20:42.916677 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:42.914728 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" podUID="4a7164ea-a62e-4d5d-b26d-adcae50ff87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:43.055017 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:43.054979 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:44.849716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:44.849687 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp" Oct 30 18:20:47.089524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:44.895695 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:47.089524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:45.987720 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g" podUID="9bd393cb-cdd9-4d66-ac02-ee4e78f0062f" containerName="payload-processing" containerID="cri-o://ecce2244bd5a2ddec6586c572836bf384490d940955559f4eeff56583c11b917" gracePeriod=25 Oct 30 18:20:47.089524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:46.037310 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn" podUID="2846f1c3-f62a-4e62-b334-d0fa80036f4a" containerName="payload-pre-processing" containerID="cri-o://8bc9cc8979470164b7a5365c499015e038c38dae7a978c591edd5c49e6a123a9" gracePeriod=25 Oct 30 18:20:47.089524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:46.231570 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:47.089524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:46.300623 4603 generic.go:356] "Generic (PLEG): container finished" podID="9bd393cb-cdd9-4d66-ac02-ee4e78f0062f" containerID="ecce2244bd5a2ddec6586c572836bf384490d940955559f4eeff56583c11b917" exitCode=0 Oct 30 18:20:47.089524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:46.300750 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g" event={"ID":"9bd393cb-cdd9-4d66-ac02-ee4e78f0062f","Type":"ContainerDied","Data":"ecce2244bd5a2ddec6586c572836bf384490d940955559f4eeff56583c11b917"} Oct 30 18:20:47.089524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:46.302746 4603 generic.go:356] "Generic (PLEG): container finished" podID="2846f1c3-f62a-4e62-b334-d0fa80036f4a" containerID="8bc9cc8979470164b7a5365c499015e038c38dae7a978c591edd5c49e6a123a9" exitCode=0 Oct 30 18:20:47.089524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:46.302892 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn" event={"ID":"2846f1c3-f62a-4e62-b334-d0fa80036f4a","Type":"ContainerDied","Data":"8bc9cc8979470164b7a5365c499015e038c38dae7a978c591edd5c49e6a123a9"} Oct 30 18:20:47.089524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:46.374135 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:47.089524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:46.389010 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g" Oct 30 18:20:47.089524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:46.428069 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bd393cb-cdd9-4d66-ac02-ee4e78f0062f-kube-api-access-gsqrt\" (UniqueName: \"kubernetes.io/projected/9bd393cb-cdd9-4d66-ac02-ee4e78f0062f-kube-api-access-gsqrt\") pod \"9bd393cb-cdd9-4d66-ac02-ee4e78f0062f\" (UID: \"9bd393cb-cdd9-4d66-ac02-ee4e78f0062f\") " Oct 30 18:20:47.089524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:46.428195 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bd393cb-cdd9-4d66-ac02-ee4e78f0062f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bd393cb-cdd9-4d66-ac02-ee4e78f0062f-plugins-config-volume\") pod \"9bd393cb-cdd9-4d66-ac02-ee4e78f0062f\" (UID: \"9bd393cb-cdd9-4d66-ac02-ee4e78f0062f\") " Oct 30 18:20:47.089524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:46.428701 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd393cb-cdd9-4d66-ac02-ee4e78f0062f-plugins-config-volume" pod "9bd393cb-cdd9-4d66-ac02-ee4e78f0062f" (UID: "9bd393cb-cdd9-4d66-ac02-ee4e78f0062f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:47.089524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:46.429017 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bd393cb-cdd9-4d66-ac02-ee4e78f0062f-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.089524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:46.431393 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd393cb-cdd9-4d66-ac02-ee4e78f0062f-kube-api-access-gsqrt" pod "9bd393cb-cdd9-4d66-ac02-ee4e78f0062f" (UID: "9bd393cb-cdd9-4d66-ac02-ee4e78f0062f"). InnerVolumeSpecName "kube-api-access-gsqrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:47.089524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:46.482314 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn" Oct 30 18:20:47.089524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:46.529683 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2846f1c3-f62a-4e62-b334-d0fa80036f4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2846f1c3-f62a-4e62-b334-d0fa80036f4a-plugins-config-volume\") pod \"2846f1c3-f62a-4e62-b334-d0fa80036f4a\" (UID: \"2846f1c3-f62a-4e62-b334-d0fa80036f4a\") " Oct 30 18:20:47.089524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:46.529734 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2846f1c3-f62a-4e62-b334-d0fa80036f4a-kube-api-access-nqz24\" (UniqueName: \"kubernetes.io/projected/2846f1c3-f62a-4e62-b334-d0fa80036f4a-kube-api-access-nqz24\") pod \"2846f1c3-f62a-4e62-b334-d0fa80036f4a\" (UID: \"2846f1c3-f62a-4e62-b334-d0fa80036f4a\") " Oct 30 18:20:47.089524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:46.530092 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2846f1c3-f62a-4e62-b334-d0fa80036f4a-plugins-config-volume" pod "2846f1c3-f62a-4e62-b334-d0fa80036f4a" (UID: "2846f1c3-f62a-4e62-b334-d0fa80036f4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:47.089524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:46.530174 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsqrt\" (UniqueName: \"kubernetes.io/projected/9bd393cb-cdd9-4d66-ac02-ee4e78f0062f-kube-api-access-gsqrt\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.089524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:46.532206 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2846f1c3-f62a-4e62-b334-d0fa80036f4a-kube-api-access-nqz24" pod "2846f1c3-f62a-4e62-b334-d0fa80036f4a" (UID: "2846f1c3-f62a-4e62-b334-d0fa80036f4a"). InnerVolumeSpecName "kube-api-access-nqz24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:47.089524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:46.640600 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2846f1c3-f62a-4e62-b334-d0fa80036f4a-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.089524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:46.640634 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqz24\" (UniqueName: \"kubernetes.io/projected/2846f1c3-f62a-4e62-b334-d0fa80036f4a-kube-api-access-nqz24\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.445088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:47.313113 4603 generic.go:356] "Generic (PLEG): container finished" podID="30b23750-56cb-4895-80a8-8e54ae819f7a" containerID="00c0b21e487800fbcf8c54e46f87001a571fa1f9f79e2f0dcbd0d555ec3b60cb" exitCode=0 Oct 30 18:20:47.445088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:47.313186 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" event={"ID":"30b23750-56cb-4895-80a8-8e54ae819f7a","Type":"ContainerDied","Data":"00c0b21e487800fbcf8c54e46f87001a571fa1f9f79e2f0dcbd0d555ec3b60cb"} Oct 30 18:20:47.445088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:47.313705 4603 scope.go:122] "RemoveContainer" containerID="00c0b21e487800fbcf8c54e46f87001a571fa1f9f79e2f0dcbd0d555ec3b60cb" Oct 30 18:20:47.445088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:47.315017 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g" Oct 30 18:20:47.445088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:47.315027 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g" event={"ID":"9bd393cb-cdd9-4d66-ac02-ee4e78f0062f","Type":"ContainerDied","Data":"d2db19db65847039adcaf036780d73566702bc3e29e6816ff4f1a52f72ba7e44"} Oct 30 18:20:47.445088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:47.315084 4603 scope.go:122] "RemoveContainer" containerID="ecce2244bd5a2ddec6586c572836bf384490d940955559f4eeff56583c11b917" Oct 30 18:20:47.445088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:47.315603 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:47.445088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:47.315688 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:47.445088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:47.315738 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:47.445088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:47.316852 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn" event={"ID":"2846f1c3-f62a-4e62-b334-d0fa80036f4a","Type":"ContainerDied","Data":"7998e599000c874ec5afb4d067188ffcc13f90e38470dd6c21a60811d4af47cb"} Oct 30 18:20:47.445088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:47.316879 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn" Oct 30 18:20:47.445088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:47.322415 4603 generic.go:356] "Generic (PLEG): container finished" podID="4a7164ea-a62e-4d5d-b26d-adcae50ff87c" containerID="9ccc70e0484185c438cedbb158cf6f81acc660fc8ef12e5eeb6ec63eea527416" exitCode=0 Oct 30 18:20:47.445088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:47.322472 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" event={"ID":"4a7164ea-a62e-4d5d-b26d-adcae50ff87c","Type":"ContainerDied","Data":"9ccc70e0484185c438cedbb158cf6f81acc660fc8ef12e5eeb6ec63eea527416"} Oct 30 18:20:47.445088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:47.322818 4603 scope.go:122] "RemoveContainer" containerID="9ccc70e0484185c438cedbb158cf6f81acc660fc8ef12e5eeb6ec63eea527416" Oct 30 18:20:47.445088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:47.331128 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:47.445088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:47.331214 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:47.445088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:47.331270 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:47.445088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:47.335178 4603 scope.go:122] "RemoveContainer" containerID="8bc9cc8979470164b7a5365c499015e038c38dae7a978c591edd5c49e6a123a9" Oct 30 18:20:47.549569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:47.545760 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn"] Oct 30 18:20:47.574759 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:47.574732 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-568d478b79-zqmrn"] Oct 30 18:20:47.623070 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:47.619643 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g"] Oct 30 18:20:47.623385 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:47.623365 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-069812-6988fd449d-w6m5g"] Oct 30 18:20:47.721609 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:47.719945 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:48.252535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:48.238546 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:48.357951 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:48.357919 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" event={"ID":"4a7164ea-a62e-4d5d-b26d-adcae50ff87c","Type":"ContainerStarted","Data":"8d437eb4d0f5bbe38d07e3d7af11ba32f57f3aea1dded6f4ed090e7db5f55cd5"} Oct 30 18:20:48.360596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:48.360568 4603 generic.go:356] "Generic (PLEG): container finished" podID="904c45e3-164e-45ee-b877-b8b565fbb7fd" containerID="017d1e8d557fa8e9be5fb47c79ee543b2df3619eb2969c6a4350433fbba9abfa" exitCode=0 Oct 30 18:20:48.360802 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:48.360767 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" event={"ID":"904c45e3-164e-45ee-b877-b8b565fbb7fd","Type":"ContainerDied","Data":"017d1e8d557fa8e9be5fb47c79ee543b2df3619eb2969c6a4350433fbba9abfa"} Oct 30 18:20:48.361292 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:48.361277 4603 scope.go:122] "RemoveContainer" containerID="017d1e8d557fa8e9be5fb47c79ee543b2df3619eb2969c6a4350433fbba9abfa" Oct 30 18:20:48.363230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:48.363199 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:48.363348 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:48.363290 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:48.363348 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:48.363335 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:48.364179 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:48.364156 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" event={"ID":"30b23750-56cb-4895-80a8-8e54ae819f7a","Type":"ContainerStarted","Data":"801170481af59c9f323d8a8822629a2f1faae32a72e425359bd87149acfddb70"} Oct 30 18:20:48.669711 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:48.520811 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:48.669711 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:48.521340 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 18:20:48.669711 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:48.521389 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" podUID="4a7164ea-a62e-4d5d-b26d-adcae50ff87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 18:20:48.669711 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:48.562026 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:20:48.933993 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:48.933903 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz"] Oct 30 18:20:48.935288 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:48.935261 4603 cpu_manager.go:414] "containerMap: removing container" podUID="9bd393cb-cdd9-4d66-ac02-ee4e78f0062f" containerName="payload-processing" Oct 30 18:20:48.935456 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:48.935441 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd393cb-cdd9-4d66-ac02-ee4e78f0062f" containerName="payload-processing" Oct 30 18:20:48.935624 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:48.935612 4603 cpu_manager.go:414] "containerMap: removing container" podUID="2846f1c3-f62a-4e62-b334-d0fa80036f4a" containerName="payload-pre-processing" Oct 30 18:20:48.935721 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:48.935710 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2846f1c3-f62a-4e62-b334-d0fa80036f4a" containerName="payload-pre-processing" Oct 30 18:20:48.936003 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:48.935989 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd393cb-cdd9-4d66-ac02-ee4e78f0062f" containerName="payload-processing" Oct 30 18:20:48.936112 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:48.936101 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="2846f1c3-f62a-4e62-b334-d0fa80036f4a" containerName="payload-pre-processing" Oct 30 18:20:48.980600 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:48.979829 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz"] Oct 30 18:20:48.980600 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:48.979932 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz" Oct 30 18:20:49.020049 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.020016 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e0bc49-dockercfg-5dsff\"" Oct 30 18:20:49.020366 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.020346 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e0bc49\"" Oct 30 18:20:49.084147 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.084100 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58abeaf7-cdae-438a-a1e3-276aa3e76393-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz\" (UID: \"58abeaf7-cdae-438a-a1e3-276aa3e76393\") " pod="openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz" Oct 30 18:20:49.084354 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.084326 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh9jk\" (UniqueName: \"kubernetes.io/projected/58abeaf7-cdae-438a-a1e3-276aa3e76393-kube-api-access-kh9jk\") pod \"payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz\" (UID: \"58abeaf7-cdae-438a-a1e3-276aa3e76393\") " pod="openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz" Oct 30 18:20:49.179663 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.179632 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:20:49.179858 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.179751 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:20:49.179858 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.179792 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" podUID="30b23750-56cb-4895-80a8-8e54ae819f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 18:20:49.186055 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.185975 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh9jk\" (UniqueName: \"kubernetes.io/projected/58abeaf7-cdae-438a-a1e3-276aa3e76393-kube-api-access-kh9jk\") pod \"payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz\" (UID: \"58abeaf7-cdae-438a-a1e3-276aa3e76393\") " pod="openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz" Oct 30 18:20:49.186201 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.186153 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58abeaf7-cdae-438a-a1e3-276aa3e76393-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz\" (UID: \"58abeaf7-cdae-438a-a1e3-276aa3e76393\") " pod="openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz" Oct 30 18:20:49.186815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.186797 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58abeaf7-cdae-438a-a1e3-276aa3e76393-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz\" (UID: \"58abeaf7-cdae-438a-a1e3-276aa3e76393\") " pod="openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz" Oct 30 18:20:49.236526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.235784 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh9jk\" (UniqueName: \"kubernetes.io/projected/58abeaf7-cdae-438a-a1e3-276aa3e76393-kube-api-access-kh9jk\") pod \"payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz\" (UID: \"58abeaf7-cdae-438a-a1e3-276aa3e76393\") " pod="openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz" Oct 30 18:20:49.357229 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.356868 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz" Oct 30 18:20:49.374836 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.374800 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" event={"ID":"904c45e3-164e-45ee-b877-b8b565fbb7fd","Type":"ContainerStarted","Data":"c24b5036c142521f0dd0323bb8c916177cfe4f0304d07a6eecbb0055150a4689"} Oct 30 18:20:49.378717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.378679 4603 generic.go:356] "Generic (PLEG): container finished" podID="3472e0a8-d76d-45af-a9f4-cae9b46712b6" containerID="914b115695314e21fb1df670d9d1d9d69a1c192160a66d27a4fa26c6ae1dafbf" exitCode=0 Oct 30 18:20:49.378924 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.378900 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" event={"ID":"3472e0a8-d76d-45af-a9f4-cae9b46712b6","Type":"ContainerDied","Data":"914b115695314e21fb1df670d9d1d9d69a1c192160a66d27a4fa26c6ae1dafbf"} Oct 30 18:20:49.379918 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.379813 4603 scope.go:122] "RemoveContainer" containerID="914b115695314e21fb1df670d9d1d9d69a1c192160a66d27a4fa26c6ae1dafbf" Oct 30 18:20:49.381726 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.381690 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:49.381835 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.381776 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:49.381835 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.381823 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:49.474493 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.472299 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 18:20:49.474493 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.472378 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" podUID="4a7164ea-a62e-4d5d-b26d-adcae50ff87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 18:20:49.553656 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.547361 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2846f1c3-f62a-4e62-b334-d0fa80036f4a" path="/var/lib/kubelet/pods/2846f1c3-f62a-4e62-b334-d0fa80036f4a/volumes" Oct 30 18:20:49.554681 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.554495 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bd393cb-cdd9-4d66-ac02-ee4e78f0062f" path="/var/lib/kubelet/pods/9bd393cb-cdd9-4d66-ac02-ee4e78f0062f/volumes" Oct 30 18:20:49.663650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.663445 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:49.996529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:49.931179 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz"] Oct 30 18:20:50.058290 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:20:50.058264 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58abeaf7_cdae_438a_a1e3_276aa3e76393.slice/crio-2c716334a4b338893c42ce8924a30e07906c313ec70b67916e87ce775179e849 WatchSource:0}: Error finding container 2c716334a4b338893c42ce8924a30e07906c313ec70b67916e87ce775179e849: Status 404 returned error can't find the container with id 2c716334a4b338893c42ce8924a30e07906c313ec70b67916e87ce775179e849 Oct 30 18:20:50.167551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:50.164860 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:20:50.167551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:50.166398 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 18:20:50.167551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:50.166434 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" podUID="904c45e3-164e-45ee-b877-b8b565fbb7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 18:20:50.178799 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:50.178774 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:20:50.178938 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:50.178847 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" podUID="30b23750-56cb-4895-80a8-8e54ae819f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 18:20:50.395170 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:50.395126 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz" event={"ID":"58abeaf7-cdae-438a-a1e3-276aa3e76393","Type":"ContainerStarted","Data":"427038c55cc4abe29cb9b28cdcd5ce3cedc61434760160b3b1384de68cdad8dc"} Oct 30 18:20:50.396540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:50.395798 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz" event={"ID":"58abeaf7-cdae-438a-a1e3-276aa3e76393","Type":"ContainerStarted","Data":"2c716334a4b338893c42ce8924a30e07906c313ec70b67916e87ce775179e849"} Oct 30 18:20:50.397371 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:50.397345 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz" Oct 30 18:20:50.401888 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:50.401861 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" event={"ID":"3472e0a8-d76d-45af-a9f4-cae9b46712b6","Type":"ContainerStarted","Data":"0bccd0424b6f1d37697cbb71196d9656ae3e70d05810e8f2716bc744e3c2db48"} Oct 30 18:20:50.481126 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:50.481061 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz" podStartSLOduration=2.481042045 podStartE2EDuration="2.481042045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:50.468360707 +0000 UTC m=+1711.183930582" watchObservedRunningTime="2026-10-30 18:20:50.481042045 +0000 UTC m=+1711.196611923" Oct 30 18:20:50.483718 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:50.483686 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 18:20:50.483843 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:50.483746 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" podUID="4a7164ea-a62e-4d5d-b26d-adcae50ff87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 18:20:50.831533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:50.829124 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:20:50.831533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:50.829274 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 18:20:50.831533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:50.829319 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" podUID="3472e0a8-d76d-45af-a9f4-cae9b46712b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 18:20:51.164595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:51.164055 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 18:20:51.164595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:51.164123 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" podUID="904c45e3-164e-45ee-b877-b8b565fbb7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 18:20:51.186529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:51.185920 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:20:51.186529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:51.185987 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" podUID="30b23750-56cb-4895-80a8-8e54ae819f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 18:20:51.471591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:51.470911 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 18:20:51.471591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:51.470973 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" podUID="4a7164ea-a62e-4d5d-b26d-adcae50ff87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 18:20:51.828530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:51.828032 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 18:20:51.828530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:51.828116 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" podUID="3472e0a8-d76d-45af-a9f4-cae9b46712b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 18:20:52.005531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:52.003625 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:52.100979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:52.100630 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:52.169620 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:52.168600 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:20:52.170347 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:52.170315 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:52.186530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:52.171363 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 18:20:52.188595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:52.188560 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" podUID="904c45e3-164e-45ee-b877-b8b565fbb7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 18:20:52.197449 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:52.197420 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:20:52.197703 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:52.197676 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" podUID="30b23750-56cb-4895-80a8-8e54ae819f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 18:20:52.235847 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:20:52.235811 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:20:52.248669 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:52.248625 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:52.473045 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:52.472959 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 18:20:52.473648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:52.473617 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" podUID="4a7164ea-a62e-4d5d-b26d-adcae50ff87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 18:20:52.750229 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:52.750130 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:52.891599 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:52.891557 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:20:52.905779 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:52.893482 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 18:20:52.905779 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:52.893555 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" podUID="3472e0a8-d76d-45af-a9f4-cae9b46712b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 18:20:53.016725 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:53.016625 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:53.170546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:53.170025 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 18:20:53.170546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:53.170093 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" podUID="904c45e3-164e-45ee-b877-b8b565fbb7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 18:20:53.181823 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:53.171936 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:53.181823 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:53.172373 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:20:53.277642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:53.229315 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:20:53.277642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:53.229375 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" podUID="30b23750-56cb-4895-80a8-8e54ae819f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 18:20:53.277642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:53.229479 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:53.277642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:53.229881 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:20:53.311482 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:53.311454 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:20:53.336225 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:53.336186 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:53.340529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:53.336968 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:20:53.477916 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:53.475012 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:53.891580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:53.757867 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:53.891941 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:53.770888 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 18:20:53.892177 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:53.892149 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" podUID="4a7164ea-a62e-4d5d-b26d-adcae50ff87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 18:20:53.892344 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:53.773045 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:20:53.893198 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:53.893171 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:54.028659 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.028613 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:54.121540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.110245 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 18:20:54.162527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.162412 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" podUID="3472e0a8-d76d-45af-a9f4-cae9b46712b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 18:20:54.215539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.190864 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:54.287528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.232406 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:54.288060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.255763 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 18:20:54.313693 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.313636 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" podUID="904c45e3-164e-45ee-b877-b8b565fbb7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 18:20:54.314076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.314033 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:54.374426 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.360804 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:54.429558 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:20:54.429448 4603 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.016s" Oct 30 18:20:54.437718 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.437691 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:20:54.457701 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.457660 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:20:54.457939 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.457923 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:20:54.458178 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.458166 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:20:54.467690 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.467663 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf" Oct 30 18:20:54.556164 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.467797 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:20:54.557456 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.557430 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" podUID="30b23750-56cb-4895-80a8-8e54ae819f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 18:20:54.557664 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.556853 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg" Oct 30 18:20:54.734022 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.733663 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 18:20:54.734022 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.733728 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" podUID="4a7164ea-a62e-4d5d-b26d-adcae50ff87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 18:20:54.785853 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.783831 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Oct 30 18:20:54.786432 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.786386 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:54.798787 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.798767 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:20:54.847687 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.847652 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 18:20:54.847951 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.847926 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" podUID="3472e0a8-d76d-45af-a9f4-cae9b46712b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 18:20:54.903655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.903613 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:54.904593 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:54.904559 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:20:55.070051 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.069982 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:55.095564 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.075889 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:20:55.219140 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.219097 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:55.227563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.227493 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 18:20:55.227790 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.227769 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" podUID="904c45e3-164e-45ee-b877-b8b565fbb7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 18:20:55.237574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.237547 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:20:55.241615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.241574 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" podUID="30b23750-56cb-4895-80a8-8e54ae819f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 18:20:55.286847 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.286820 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:20:55.319331 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.319274 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:55.320400 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.320312 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:20:55.343002 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.320654 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:20:55.368546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.368486 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:55.425082 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.425043 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:20:55.474724 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.474689 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:55.500816 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.500685 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Oct 30 18:20:55.562754 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.562717 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:20:55.564705 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.564689 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:20:55.566214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.565998 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:20:55.587797 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.587708 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:20:55.589921 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.589735 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:20:55.592547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.592528 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:20:55.638024 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.637977 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:20:55.639301 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.639281 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:20:55.652996 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.652963 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:20:55.854337 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.854249 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 18:20:55.854621 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:55.854578 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" podUID="3472e0a8-d76d-45af-a9f4-cae9b46712b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 18:20:56.167980 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:56.167889 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 18:20:56.168262 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:56.168232 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" podUID="904c45e3-164e-45ee-b877-b8b565fbb7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 18:20:56.224231 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:56.224188 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:20:56.253068 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:56.253032 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:20:56.830528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:56.829939 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 18:20:56.830528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:56.830004 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" podUID="3472e0a8-d76d-45af-a9f4-cae9b46712b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 18:20:57.173178 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:57.173098 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:20:57.174900 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:57.174874 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:20:57.753337 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:57.753304 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:57.868085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:57.868053 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:20:57.871542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:57.871481 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:20:59.430646 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:20:59.430606 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:01.106907 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:01.106867 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:02.531138 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:02.531106 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz" Oct 30 18:21:03.045634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:03.045602 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:03.133133 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:03.133101 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz"] Oct 30 18:21:03.133626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:03.133594 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz" podUID="b5425f00-684b-4f8c-a9fb-7b7e534f66ed" containerName="payload-pre-processing" containerID="cri-o://0b8f80ac74b61a7ba131da0ed23029f798d9f7a6f5b8c1e05a71a83b0908df14" gracePeriod=30 Oct 30 18:21:03.222268 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:03.222230 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:03.367438 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:03.367400 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:03.762578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:03.757392 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:03.798829 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:03.798799 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz" Oct 30 18:21:03.803798 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:03.803766 4603 generic.go:356] "Generic (PLEG): container finished" podID="b5425f00-684b-4f8c-a9fb-7b7e534f66ed" containerID="0b8f80ac74b61a7ba131da0ed23029f798d9f7a6f5b8c1e05a71a83b0908df14" exitCode=0 Oct 30 18:21:03.804136 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:03.804111 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz" event={"ID":"b5425f00-684b-4f8c-a9fb-7b7e534f66ed","Type":"ContainerDied","Data":"0b8f80ac74b61a7ba131da0ed23029f798d9f7a6f5b8c1e05a71a83b0908df14"} Oct 30 18:21:03.804342 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:03.804322 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz" event={"ID":"b5425f00-684b-4f8c-a9fb-7b7e534f66ed","Type":"ContainerDied","Data":"3bb4f52e441a48ef076240214b82908a4bc745fcc0f3437ee119f90ad63ad64b"} Oct 30 18:21:03.804465 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:03.804451 4603 scope.go:122] "RemoveContainer" containerID="0b8f80ac74b61a7ba131da0ed23029f798d9f7a6f5b8c1e05a71a83b0908df14" Oct 30 18:21:03.842131 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:03.842106 4603 scope.go:122] "RemoveContainer" containerID="0b8f80ac74b61a7ba131da0ed23029f798d9f7a6f5b8c1e05a71a83b0908df14" Oct 30 18:21:03.843193 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:21:03.843168 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b8f80ac74b61a7ba131da0ed23029f798d9f7a6f5b8c1e05a71a83b0908df14\": container with ID starting with 0b8f80ac74b61a7ba131da0ed23029f798d9f7a6f5b8c1e05a71a83b0908df14 not found: ID does not exist" containerID="0b8f80ac74b61a7ba131da0ed23029f798d9f7a6f5b8c1e05a71a83b0908df14" Oct 30 18:21:03.843381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:03.843350 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b8f80ac74b61a7ba131da0ed23029f798d9f7a6f5b8c1e05a71a83b0908df14"} err="failed to get container status \"0b8f80ac74b61a7ba131da0ed23029f798d9f7a6f5b8c1e05a71a83b0908df14\": rpc error: code = NotFound desc = could not find container \"0b8f80ac74b61a7ba131da0ed23029f798d9f7a6f5b8c1e05a71a83b0908df14\": container with ID starting with 0b8f80ac74b61a7ba131da0ed23029f798d9f7a6f5b8c1e05a71a83b0908df14 not found: ID does not exist" Oct 30 18:21:03.903379 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:03.903345 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:03.943646 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:03.934677 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5425f00-684b-4f8c-a9fb-7b7e534f66ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5425f00-684b-4f8c-a9fb-7b7e534f66ed-plugins-config-volume\") pod \"b5425f00-684b-4f8c-a9fb-7b7e534f66ed\" (UID: \"b5425f00-684b-4f8c-a9fb-7b7e534f66ed\") " Oct 30 18:21:03.944180 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:03.944160 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5425f00-684b-4f8c-a9fb-7b7e534f66ed-kube-api-access-wjkh5\" (UniqueName: \"kubernetes.io/projected/b5425f00-684b-4f8c-a9fb-7b7e534f66ed-kube-api-access-wjkh5\") pod \"b5425f00-684b-4f8c-a9fb-7b7e534f66ed\" (UID: \"b5425f00-684b-4f8c-a9fb-7b7e534f66ed\") " Oct 30 18:21:03.945025 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:03.940691 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5425f00-684b-4f8c-a9fb-7b7e534f66ed-plugins-config-volume" pod "b5425f00-684b-4f8c-a9fb-7b7e534f66ed" (UID: "b5425f00-684b-4f8c-a9fb-7b7e534f66ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:03.945899 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:03.945880 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5425f00-684b-4f8c-a9fb-7b7e534f66ed-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:03.947374 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:03.947345 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5425f00-684b-4f8c-a9fb-7b7e534f66ed-kube-api-access-wjkh5" pod "b5425f00-684b-4f8c-a9fb-7b7e534f66ed" (UID: "b5425f00-684b-4f8c-a9fb-7b7e534f66ed"). InnerVolumeSpecName "kube-api-access-wjkh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:04.054940 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:04.052726 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjkh5\" (UniqueName: \"kubernetes.io/projected/b5425f00-684b-4f8c-a9fb-7b7e534f66ed-kube-api-access-wjkh5\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:04.076294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:04.076015 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:04.255972 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:04.255946 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:04.811757 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:04.811727 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz" Oct 30 18:21:04.863254 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:04.863226 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz"] Oct 30 18:21:04.879738 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:04.879707 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-747dfd9bf9-cj7zz"] Oct 30 18:21:05.256189 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:05.256107 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:05.422552 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:05.411999 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:05.534783 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:05.534740 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5425f00-684b-4f8c-a9fb-7b7e534f66ed" path="/var/lib/kubelet/pods/b5425f00-684b-4f8c-a9fb-7b7e534f66ed/volumes" Oct 30 18:21:05.876881 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:05.876803 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f"] Oct 30 18:21:05.878654 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:05.878619 4603 cpu_manager.go:414] "containerMap: removing container" podUID="b5425f00-684b-4f8c-a9fb-7b7e534f66ed" containerName="payload-pre-processing" Oct 30 18:21:05.878865 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:05.878848 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5425f00-684b-4f8c-a9fb-7b7e534f66ed" containerName="payload-pre-processing" Oct 30 18:21:05.879304 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:05.879280 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5425f00-684b-4f8c-a9fb-7b7e534f66ed" containerName="payload-pre-processing" Oct 30 18:21:05.883676 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:05.883636 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f" Oct 30 18:21:05.890486 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:05.890461 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f"] Oct 30 18:21:05.924649 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:05.924618 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-069812\"" Oct 30 18:21:05.976427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:05.967634 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-069812-dockercfg-9t6ft\"" Oct 30 18:21:06.085539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:06.082272 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f\" (UID: \"fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f" Oct 30 18:21:06.085539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:06.082371 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppwcm\" (UniqueName: \"kubernetes.io/projected/fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba-kube-api-access-ppwcm\") pod \"payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f\" (UID: \"fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f" Oct 30 18:21:06.184315 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:06.184224 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f\" (UID: \"fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f" Oct 30 18:21:06.184516 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:06.184320 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppwcm\" (UniqueName: \"kubernetes.io/projected/fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba-kube-api-access-ppwcm\") pod \"payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f\" (UID: \"fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f" Oct 30 18:21:06.184967 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:06.184947 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f\" (UID: \"fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f" Oct 30 18:21:06.207676 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:06.207651 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppwcm\" (UniqueName: \"kubernetes.io/projected/fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba-kube-api-access-ppwcm\") pod \"payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f\" (UID: \"fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f" Oct 30 18:21:06.317606 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:06.317559 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f" Oct 30 18:21:06.490378 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:06.490089 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:06.667619 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:06.667593 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f"] Oct 30 18:21:06.668761 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:21:06.668733 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa3d340a_2ea0_4c55_8a5f_ed1b5c8f3bba.slice/crio-3cdaeb3c408af06f0cd7280290c0c654d71d3e95f0531b6a0683f1cf2c3f10ce WatchSource:0}: Error finding container 3cdaeb3c408af06f0cd7280290c0c654d71d3e95f0531b6a0683f1cf2c3f10ce: Status 404 returned error can't find the container with id 3cdaeb3c408af06f0cd7280290c0c654d71d3e95f0531b6a0683f1cf2c3f10ce Oct 30 18:21:06.879829 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:06.879797 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f" event={"ID":"fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba","Type":"ContainerStarted","Data":"1b6db1275eb9474b1eaf26105e24fe04c93c63ea058ed02013fe45b47d4c7d03"} Oct 30 18:21:06.880298 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:06.879854 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f" Oct 30 18:21:06.880298 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:06.879871 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f" event={"ID":"fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba","Type":"ContainerStarted","Data":"3cdaeb3c408af06f0cd7280290c0c654d71d3e95f0531b6a0683f1cf2c3f10ce"} Oct 30 18:21:06.921548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:06.921485 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f" podStartSLOduration=1.921469425 podStartE2EDuration="1.921469425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:06.919075979 +0000 UTC m=+1727.634645849" watchObservedRunningTime="2026-10-30 18:21:06.921469425 +0000 UTC m=+1727.637039299" Oct 30 18:21:08.374757 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:08.374399 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:09.316321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:09.316285 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:09.455866 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:09.455836 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf"] Oct 30 18:21:09.492641 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:09.492611 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg"] Oct 30 18:21:09.635332 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:09.635257 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:09.800533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:09.786054 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:10.673472 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:10.673439 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:10.818093 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:10.818055 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:10.935303 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:10.935187 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp"] Oct 30 18:21:10.935570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:10.935540 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" podUID="18da6086-657d-4917-9fcb-c8de74e7f7b4" containerName="istio-proxy" containerID="cri-o://20f2d6ebcae7202fb4c21de11165d87f422f0074107df82dca9b64d52c516f41" gracePeriod=30 Oct 30 18:21:10.963805 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:10.963780 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:11.129529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:11.123494 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:12.513769 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:12.513722 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:14.158541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.158482 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:14.456696 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.456644 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf" podUID="6fa3594b-6dab-4f71-a8c1-183151163cfb" containerName="payload-processing" containerID="cri-o://c08244d35a17e2933f026ea8da610ca34689a82f4cfb56e1b1a3c1c802209422" gracePeriod=25 Oct 30 18:21:14.493836 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.493771 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg" podUID="ea551dc1-ef6e-4a74-a118-bef1b0f70a57" containerName="payload-pre-processing" containerID="cri-o://f9d01f34e96b92680ab80d535949fec53341e396e02ba2a0050f00c7f086d1f2" gracePeriod=25 Oct 30 18:21:14.779733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.779701 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf" Oct 30 18:21:14.818451 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.818428 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg" Oct 30 18:21:14.877796 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.877769 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fa3594b-6dab-4f71-a8c1-183151163cfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa3594b-6dab-4f71-a8c1-183151163cfb-plugins-config-volume\") pod \"6fa3594b-6dab-4f71-a8c1-183151163cfb\" (UID: \"6fa3594b-6dab-4f71-a8c1-183151163cfb\") " Oct 30 18:21:14.877961 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.877819 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea551dc1-ef6e-4a74-a118-bef1b0f70a57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea551dc1-ef6e-4a74-a118-bef1b0f70a57-plugins-config-volume\") pod \"ea551dc1-ef6e-4a74-a118-bef1b0f70a57\" (UID: \"ea551dc1-ef6e-4a74-a118-bef1b0f70a57\") " Oct 30 18:21:14.878112 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.878071 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa3594b-6dab-4f71-a8c1-183151163cfb-kube-api-access-45x2d\" (UniqueName: \"kubernetes.io/projected/6fa3594b-6dab-4f71-a8c1-183151163cfb-kube-api-access-45x2d\") pod \"6fa3594b-6dab-4f71-a8c1-183151163cfb\" (UID: \"6fa3594b-6dab-4f71-a8c1-183151163cfb\") " Oct 30 18:21:14.878192 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.878169 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea551dc1-ef6e-4a74-a118-bef1b0f70a57-kube-api-access-plfnt\" (UniqueName: \"kubernetes.io/projected/ea551dc1-ef6e-4a74-a118-bef1b0f70a57-kube-api-access-plfnt\") pod \"ea551dc1-ef6e-4a74-a118-bef1b0f70a57\" (UID: \"ea551dc1-ef6e-4a74-a118-bef1b0f70a57\") " Oct 30 18:21:14.878259 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.878232 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fa3594b-6dab-4f71-a8c1-183151163cfb-plugins-config-volume" pod "6fa3594b-6dab-4f71-a8c1-183151163cfb" (UID: "6fa3594b-6dab-4f71-a8c1-183151163cfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:14.878304 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.878248 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea551dc1-ef6e-4a74-a118-bef1b0f70a57-plugins-config-volume" pod "ea551dc1-ef6e-4a74-a118-bef1b0f70a57" (UID: "ea551dc1-ef6e-4a74-a118-bef1b0f70a57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:14.878847 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.878814 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa3594b-6dab-4f71-a8c1-183151163cfb-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:14.878847 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.878834 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea551dc1-ef6e-4a74-a118-bef1b0f70a57-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:14.881219 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.881196 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea551dc1-ef6e-4a74-a118-bef1b0f70a57-kube-api-access-plfnt" pod "ea551dc1-ef6e-4a74-a118-bef1b0f70a57" (UID: "ea551dc1-ef6e-4a74-a118-bef1b0f70a57"). InnerVolumeSpecName "kube-api-access-plfnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:14.881347 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.881225 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa3594b-6dab-4f71-a8c1-183151163cfb-kube-api-access-45x2d" pod "6fa3594b-6dab-4f71-a8c1-183151163cfb" (UID: "6fa3594b-6dab-4f71-a8c1-183151163cfb"). InnerVolumeSpecName "kube-api-access-45x2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:14.980293 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.980264 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45x2d\" (UniqueName: \"kubernetes.io/projected/6fa3594b-6dab-4f71-a8c1-183151163cfb-kube-api-access-45x2d\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:14.980293 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.980290 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plfnt\" (UniqueName: \"kubernetes.io/projected/ea551dc1-ef6e-4a74-a118-bef1b0f70a57-kube-api-access-plfnt\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:14.986473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.986402 4603 generic.go:356] "Generic (PLEG): container finished" podID="6fa3594b-6dab-4f71-a8c1-183151163cfb" containerID="c08244d35a17e2933f026ea8da610ca34689a82f4cfb56e1b1a3c1c802209422" exitCode=0 Oct 30 18:21:14.986473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.986453 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf" Oct 30 18:21:14.986644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.986492 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf" event={"ID":"6fa3594b-6dab-4f71-a8c1-183151163cfb","Type":"ContainerDied","Data":"c08244d35a17e2933f026ea8da610ca34689a82f4cfb56e1b1a3c1c802209422"} Oct 30 18:21:14.986644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.986556 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf" event={"ID":"6fa3594b-6dab-4f71-a8c1-183151163cfb","Type":"ContainerDied","Data":"d894bf625f6bb63c5c1a863563e71d6a791e5f0f0813513e58101590d9aa7e18"} Oct 30 18:21:14.986644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.986580 4603 scope.go:122] "RemoveContainer" containerID="c08244d35a17e2933f026ea8da610ca34689a82f4cfb56e1b1a3c1c802209422" Oct 30 18:21:14.988200 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.988171 4603 generic.go:356] "Generic (PLEG): container finished" podID="ea551dc1-ef6e-4a74-a118-bef1b0f70a57" containerID="f9d01f34e96b92680ab80d535949fec53341e396e02ba2a0050f00c7f086d1f2" exitCode=0 Oct 30 18:21:14.988315 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.988202 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg" Oct 30 18:21:14.988315 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.988250 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg" event={"ID":"ea551dc1-ef6e-4a74-a118-bef1b0f70a57","Type":"ContainerDied","Data":"f9d01f34e96b92680ab80d535949fec53341e396e02ba2a0050f00c7f086d1f2"} Oct 30 18:21:14.988315 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.988309 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg" event={"ID":"ea551dc1-ef6e-4a74-a118-bef1b0f70a57","Type":"ContainerDied","Data":"93004c52b72962190bc9b20925c1bd4cd04814638cc78b5aad376fddce7e996c"} Oct 30 18:21:14.999556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.999539 4603 scope.go:122] "RemoveContainer" containerID="c08244d35a17e2933f026ea8da610ca34689a82f4cfb56e1b1a3c1c802209422" Oct 30 18:21:14.999843 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:21:14.999820 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c08244d35a17e2933f026ea8da610ca34689a82f4cfb56e1b1a3c1c802209422\": container with ID starting with c08244d35a17e2933f026ea8da610ca34689a82f4cfb56e1b1a3c1c802209422 not found: ID does not exist" containerID="c08244d35a17e2933f026ea8da610ca34689a82f4cfb56e1b1a3c1c802209422" Oct 30 18:21:14.999928 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.999851 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c08244d35a17e2933f026ea8da610ca34689a82f4cfb56e1b1a3c1c802209422"} err="failed to get container status \"c08244d35a17e2933f026ea8da610ca34689a82f4cfb56e1b1a3c1c802209422\": rpc error: code = NotFound desc = could not find container \"c08244d35a17e2933f026ea8da610ca34689a82f4cfb56e1b1a3c1c802209422\": container with ID starting with c08244d35a17e2933f026ea8da610ca34689a82f4cfb56e1b1a3c1c802209422 not found: ID does not exist" Oct 30 18:21:14.999928 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:14.999875 4603 scope.go:122] "RemoveContainer" containerID="f9d01f34e96b92680ab80d535949fec53341e396e02ba2a0050f00c7f086d1f2" Oct 30 18:21:15.010111 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:15.010089 4603 scope.go:122] "RemoveContainer" containerID="f9d01f34e96b92680ab80d535949fec53341e396e02ba2a0050f00c7f086d1f2" Oct 30 18:21:15.010435 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:21:15.010410 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d01f34e96b92680ab80d535949fec53341e396e02ba2a0050f00c7f086d1f2\": container with ID starting with f9d01f34e96b92680ab80d535949fec53341e396e02ba2a0050f00c7f086d1f2 not found: ID does not exist" containerID="f9d01f34e96b92680ab80d535949fec53341e396e02ba2a0050f00c7f086d1f2" Oct 30 18:21:15.010553 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:15.010446 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d01f34e96b92680ab80d535949fec53341e396e02ba2a0050f00c7f086d1f2"} err="failed to get container status \"f9d01f34e96b92680ab80d535949fec53341e396e02ba2a0050f00c7f086d1f2\": rpc error: code = NotFound desc = could not find container \"f9d01f34e96b92680ab80d535949fec53341e396e02ba2a0050f00c7f086d1f2\": container with ID starting with f9d01f34e96b92680ab80d535949fec53341e396e02ba2a0050f00c7f086d1f2 not found: ID does not exist" Oct 30 18:21:15.039704 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:15.039686 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf"] Oct 30 18:21:15.043154 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:15.043135 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-394a31-75fc7b589b-jczbf"] Oct 30 18:21:15.077748 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:15.077720 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg"] Oct 30 18:21:15.081486 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:15.081465 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-6d66c9c58c-vsvxg"] Oct 30 18:21:15.411207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:15.411174 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fa3594b-6dab-4f71-a8c1-183151163cfb" path="/var/lib/kubelet/pods/6fa3594b-6dab-4f71-a8c1-183151163cfb/volumes" Oct 30 18:21:15.411605 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:15.411590 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea551dc1-ef6e-4a74-a118-bef1b0f70a57" path="/var/lib/kubelet/pods/ea551dc1-ef6e-4a74-a118-bef1b0f70a57/volumes" Oct 30 18:21:15.774664 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:15.774635 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:16.245917 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.245891 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:21:16.299917 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.299245 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-data\") pod \"18da6086-657d-4917-9fcb-c8de74e7f7b4\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " Oct 30 18:21:16.299917 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.299295 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-credential-socket\") pod \"18da6086-657d-4917-9fcb-c8de74e7f7b4\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " Oct 30 18:21:16.299917 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.299325 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-workload-socket\") pod \"18da6086-657d-4917-9fcb-c8de74e7f7b4\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " Oct 30 18:21:16.299917 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.299386 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18da6086-657d-4917-9fcb-c8de74e7f7b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18da6086-657d-4917-9fcb-c8de74e7f7b4-istiod-ca-cert\") pod \"18da6086-657d-4917-9fcb-c8de74e7f7b4\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " Oct 30 18:21:16.299917 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.299435 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18da6086-657d-4917-9fcb-c8de74e7f7b4-kube-api-access-chltx\" (UniqueName: \"kubernetes.io/projected/18da6086-657d-4917-9fcb-c8de74e7f7b4-kube-api-access-chltx\") pod \"18da6086-657d-4917-9fcb-c8de74e7f7b4\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " Oct 30 18:21:16.299917 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.299481 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-podinfo\") pod \"18da6086-657d-4917-9fcb-c8de74e7f7b4\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " Oct 30 18:21:16.299917 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.299548 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-data" pod "18da6086-657d-4917-9fcb-c8de74e7f7b4" (UID: "18da6086-657d-4917-9fcb-c8de74e7f7b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:16.299917 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.299603 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-workload-certs\") pod \"18da6086-657d-4917-9fcb-c8de74e7f7b4\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " Oct 30 18:21:16.299917 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.299693 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-token\" (UniqueName: \"kubernetes.io/projected/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-token\") pod \"18da6086-657d-4917-9fcb-c8de74e7f7b4\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " Oct 30 18:21:16.299917 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.299778 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-envoy\") pod \"18da6086-657d-4917-9fcb-c8de74e7f7b4\" (UID: \"18da6086-657d-4917-9fcb-c8de74e7f7b4\") " Oct 30 18:21:16.300913 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.300349 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-workload-certs" pod "18da6086-657d-4917-9fcb-c8de74e7f7b4" (UID: "18da6086-657d-4917-9fcb-c8de74e7f7b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:16.300913 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.300638 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:16.300913 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.300660 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:16.300913 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.300729 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-workload-socket" pod "18da6086-657d-4917-9fcb-c8de74e7f7b4" (UID: "18da6086-657d-4917-9fcb-c8de74e7f7b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:16.300913 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.300991 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-credential-socket" pod "18da6086-657d-4917-9fcb-c8de74e7f7b4" (UID: "18da6086-657d-4917-9fcb-c8de74e7f7b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:16.302435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.302144 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18da6086-657d-4917-9fcb-c8de74e7f7b4-istiod-ca-cert" pod "18da6086-657d-4917-9fcb-c8de74e7f7b4" (UID: "18da6086-657d-4917-9fcb-c8de74e7f7b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:16.303243 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.303057 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-envoy" pod "18da6086-657d-4917-9fcb-c8de74e7f7b4" (UID: "18da6086-657d-4917-9fcb-c8de74e7f7b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:16.303353 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.303259 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-podinfo" pod "18da6086-657d-4917-9fcb-c8de74e7f7b4" (UID: "18da6086-657d-4917-9fcb-c8de74e7f7b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:21:16.305776 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.305710 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18da6086-657d-4917-9fcb-c8de74e7f7b4-kube-api-access-chltx" pod "18da6086-657d-4917-9fcb-c8de74e7f7b4" (UID: "18da6086-657d-4917-9fcb-c8de74e7f7b4"). InnerVolumeSpecName "kube-api-access-chltx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:16.305776 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.305745 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-token" pod "18da6086-657d-4917-9fcb-c8de74e7f7b4" (UID: "18da6086-657d-4917-9fcb-c8de74e7f7b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:16.306103 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.306084 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:16.445099 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.444853 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:16.445099 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.444883 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:16.445099 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.445009 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18da6086-657d-4917-9fcb-c8de74e7f7b4-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:16.445099 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.445030 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chltx\" (UniqueName: \"kubernetes.io/projected/18da6086-657d-4917-9fcb-c8de74e7f7b4-kube-api-access-chltx\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:16.445099 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.445044 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:16.445099 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.445059 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:16.445099 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.445073 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18da6086-657d-4917-9fcb-c8de74e7f7b4-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:16.482113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:16.471601 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:17.005655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:17.005621 4603 generic.go:356] "Generic (PLEG): container finished" podID="18da6086-657d-4917-9fcb-c8de74e7f7b4" containerID="20f2d6ebcae7202fb4c21de11165d87f422f0074107df82dca9b64d52c516f41" exitCode=0 Oct 30 18:21:17.005820 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:17.005674 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" Oct 30 18:21:17.005820 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:17.005706 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" event={"ID":"18da6086-657d-4917-9fcb-c8de74e7f7b4","Type":"ContainerDied","Data":"20f2d6ebcae7202fb4c21de11165d87f422f0074107df82dca9b64d52c516f41"} Oct 30 18:21:17.005820 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:17.005767 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp" event={"ID":"18da6086-657d-4917-9fcb-c8de74e7f7b4","Type":"ContainerDied","Data":"67e8ca294440e0b44014a76749480935ecf333724f4746e22ec7d43cff798aaf"} Oct 30 18:21:17.005820 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:17.005800 4603 scope.go:122] "RemoveContainer" containerID="20f2d6ebcae7202fb4c21de11165d87f422f0074107df82dca9b64d52c516f41" Oct 30 18:21:17.020962 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:17.020944 4603 scope.go:122] "RemoveContainer" containerID="3be733c96c1fbacee466c4189709eeeecf86becd478b651780424ad21f4e7a24" Oct 30 18:21:17.113413 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:17.113375 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp"] Oct 30 18:21:17.123963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:17.123929 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-064408-openshift-default-747f9dd5ff-pd7lp"] Oct 30 18:21:17.133918 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:17.133896 4603 scope.go:122] "RemoveContainer" containerID="20f2d6ebcae7202fb4c21de11165d87f422f0074107df82dca9b64d52c516f41" Oct 30 18:21:17.134297 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:21:17.134274 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20f2d6ebcae7202fb4c21de11165d87f422f0074107df82dca9b64d52c516f41\": container with ID starting with 20f2d6ebcae7202fb4c21de11165d87f422f0074107df82dca9b64d52c516f41 not found: ID does not exist" containerID="20f2d6ebcae7202fb4c21de11165d87f422f0074107df82dca9b64d52c516f41" Oct 30 18:21:17.134415 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:17.134311 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20f2d6ebcae7202fb4c21de11165d87f422f0074107df82dca9b64d52c516f41"} err="failed to get container status \"20f2d6ebcae7202fb4c21de11165d87f422f0074107df82dca9b64d52c516f41\": rpc error: code = NotFound desc = could not find container \"20f2d6ebcae7202fb4c21de11165d87f422f0074107df82dca9b64d52c516f41\": container with ID starting with 20f2d6ebcae7202fb4c21de11165d87f422f0074107df82dca9b64d52c516f41 not found: ID does not exist" Oct 30 18:21:17.134415 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:17.134337 4603 scope.go:122] "RemoveContainer" containerID="3be733c96c1fbacee466c4189709eeeecf86becd478b651780424ad21f4e7a24" Oct 30 18:21:17.134695 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:21:17.134679 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3be733c96c1fbacee466c4189709eeeecf86becd478b651780424ad21f4e7a24\": container with ID starting with 3be733c96c1fbacee466c4189709eeeecf86becd478b651780424ad21f4e7a24 not found: ID does not exist" containerID="3be733c96c1fbacee466c4189709eeeecf86becd478b651780424ad21f4e7a24" Oct 30 18:21:17.134740 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:17.134700 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3be733c96c1fbacee466c4189709eeeecf86becd478b651780424ad21f4e7a24"} err="failed to get container status \"3be733c96c1fbacee466c4189709eeeecf86becd478b651780424ad21f4e7a24\": rpc error: code = NotFound desc = could not find container \"3be733c96c1fbacee466c4189709eeeecf86becd478b651780424ad21f4e7a24\": container with ID starting with 3be733c96c1fbacee466c4189709eeeecf86becd478b651780424ad21f4e7a24 not found: ID does not exist" Oct 30 18:21:17.500531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:17.496614 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18da6086-657d-4917-9fcb-c8de74e7f7b4" path="/var/lib/kubelet/pods/18da6086-657d-4917-9fcb-c8de74e7f7b4/volumes" Oct 30 18:21:17.935600 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:17.935568 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f" Oct 30 18:21:18.063190 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:18.063153 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:19.898526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:19.890997 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:21.910670 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:21.910641 4603 scope.go:122] "RemoveContainer" containerID="d8540aad3df7dc33d7a250392ce1302f71e36d23a2dec67321d6cb14725c4e98" Oct 30 18:21:21.934254 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:21:21.934173 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:21:21.956457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:21.956428 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:23.662246 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:23.662204 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:25.067792 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:25.067758 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:25.215120 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:25.215090 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:25.345935 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:25.345860 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:25.694967 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:25.694780 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:25.841990 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:25.841255 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:26.346989 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:26.346955 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:27.714161 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:27.714126 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:29.348542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:29.348482 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:31.487401 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:31.487364 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:32.205156 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:32.205081 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:32.349020 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:32.348987 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:32.498850 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:32.498770 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl"] Oct 30 18:21:32.499317 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:32.499030 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" podUID="f7486973-71c0-4102-9f74-c814bf601ec5" containerName="istio-proxy" containerID="cri-o://6afbbbbda0c22724bb992044b34f8f4c8a558230f5a5ba229d5ed89d1adffa0a" gracePeriod=30 Oct 30 18:21:32.583715 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:32.583687 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:33.853976 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:33.853943 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:35.240017 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:35.239984 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:35.526800 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:35.526667 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:36.404086 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:36.404031 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:37.780659 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.780638 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:21:37.925177 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.925148 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-credential-socket\") pod \"f7486973-71c0-4102-9f74-c814bf601ec5\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " Oct 30 18:21:37.925364 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.925194 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-workload-socket\") pod \"f7486973-71c0-4102-9f74-c814bf601ec5\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " Oct 30 18:21:37.925364 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.925219 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-istio-envoy\") pod \"f7486973-71c0-4102-9f74-c814bf601ec5\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " Oct 30 18:21:37.925364 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.925237 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7486973-71c0-4102-9f74-c814bf601ec5-kube-api-access-qwgj5\" (UniqueName: \"kubernetes.io/projected/f7486973-71c0-4102-9f74-c814bf601ec5-kube-api-access-qwgj5\") pod \"f7486973-71c0-4102-9f74-c814bf601ec5\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " Oct 30 18:21:37.925364 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.925274 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7486973-71c0-4102-9f74-c814bf601ec5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7486973-71c0-4102-9f74-c814bf601ec5-istio-podinfo\") pod \"f7486973-71c0-4102-9f74-c814bf601ec5\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " Oct 30 18:21:37.925364 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.925314 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7486973-71c0-4102-9f74-c814bf601ec5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7486973-71c0-4102-9f74-c814bf601ec5-istiod-ca-cert\") pod \"f7486973-71c0-4102-9f74-c814bf601ec5\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " Oct 30 18:21:37.925779 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.925439 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-credential-socket" pod "f7486973-71c0-4102-9f74-c814bf601ec5" (UID: "f7486973-71c0-4102-9f74-c814bf601ec5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:37.925779 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.925473 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7486973-71c0-4102-9f74-c814bf601ec5-istio-token\" (UniqueName: \"kubernetes.io/projected/f7486973-71c0-4102-9f74-c814bf601ec5-istio-token\") pod \"f7486973-71c0-4102-9f74-c814bf601ec5\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " Oct 30 18:21:37.925779 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.925537 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-workload-socket" pod "f7486973-71c0-4102-9f74-c814bf601ec5" (UID: "f7486973-71c0-4102-9f74-c814bf601ec5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:37.925779 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.925738 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-workload-certs\") pod \"f7486973-71c0-4102-9f74-c814bf601ec5\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " Oct 30 18:21:37.926006 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.925875 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-istio-data\") pod \"f7486973-71c0-4102-9f74-c814bf601ec5\" (UID: \"f7486973-71c0-4102-9f74-c814bf601ec5\") " Oct 30 18:21:37.926077 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.925982 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7486973-71c0-4102-9f74-c814bf601ec5-istiod-ca-cert" pod "f7486973-71c0-4102-9f74-c814bf601ec5" (UID: "f7486973-71c0-4102-9f74-c814bf601ec5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:37.926210 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.926142 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-workload-certs" pod "f7486973-71c0-4102-9f74-c814bf601ec5" (UID: "f7486973-71c0-4102-9f74-c814bf601ec5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:37.926434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.926410 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-istio-data" pod "f7486973-71c0-4102-9f74-c814bf601ec5" (UID: "f7486973-71c0-4102-9f74-c814bf601ec5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:37.926875 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.926851 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:37.926975 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.926880 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:37.926975 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.926894 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:37.926975 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.926910 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:37.926975 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.926924 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7486973-71c0-4102-9f74-c814bf601ec5-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:37.927660 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.927631 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-istio-envoy" pod "f7486973-71c0-4102-9f74-c814bf601ec5" (UID: "f7486973-71c0-4102-9f74-c814bf601ec5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:37.927784 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.927767 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7486973-71c0-4102-9f74-c814bf601ec5-kube-api-access-qwgj5" pod "f7486973-71c0-4102-9f74-c814bf601ec5" (UID: "f7486973-71c0-4102-9f74-c814bf601ec5"). InnerVolumeSpecName "kube-api-access-qwgj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:37.927892 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.927871 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7486973-71c0-4102-9f74-c814bf601ec5-istio-podinfo" pod "f7486973-71c0-4102-9f74-c814bf601ec5" (UID: "f7486973-71c0-4102-9f74-c814bf601ec5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:21:37.927971 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:37.927957 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7486973-71c0-4102-9f74-c814bf601ec5-istio-token" pod "f7486973-71c0-4102-9f74-c814bf601ec5" (UID: "f7486973-71c0-4102-9f74-c814bf601ec5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:38.028010 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:38.027984 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7486973-71c0-4102-9f74-c814bf601ec5-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:38.028010 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:38.028008 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwgj5\" (UniqueName: \"kubernetes.io/projected/f7486973-71c0-4102-9f74-c814bf601ec5-kube-api-access-qwgj5\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:38.028151 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:38.028019 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7486973-71c0-4102-9f74-c814bf601ec5-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:38.028151 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:38.028029 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7486973-71c0-4102-9f74-c814bf601ec5-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:38.179103 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:38.179035 4603 generic.go:356] "Generic (PLEG): container finished" podID="f7486973-71c0-4102-9f74-c814bf601ec5" containerID="6afbbbbda0c22724bb992044b34f8f4c8a558230f5a5ba229d5ed89d1adffa0a" exitCode=0 Oct 30 18:21:38.179103 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:38.179087 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" Oct 30 18:21:38.179265 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:38.179106 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" event={"ID":"f7486973-71c0-4102-9f74-c814bf601ec5","Type":"ContainerDied","Data":"6afbbbbda0c22724bb992044b34f8f4c8a558230f5a5ba229d5ed89d1adffa0a"} Oct 30 18:21:38.179265 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:38.179148 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl" event={"ID":"f7486973-71c0-4102-9f74-c814bf601ec5","Type":"ContainerDied","Data":"6b0b96201bec7c5b8b3215fc38161abed3e3bcdf7dcd779f80f98b411e987027"} Oct 30 18:21:38.179265 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:38.179171 4603 scope.go:122] "RemoveContainer" containerID="6afbbbbda0c22724bb992044b34f8f4c8a558230f5a5ba229d5ed89d1adffa0a" Oct 30 18:21:38.190096 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:38.190081 4603 scope.go:122] "RemoveContainer" containerID="c7b288f582bf7a4698353f293951d0b773c2be98ba32cb1f8abb2aa0de178bc3" Oct 30 18:21:38.199075 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:38.199059 4603 scope.go:122] "RemoveContainer" containerID="6afbbbbda0c22724bb992044b34f8f4c8a558230f5a5ba229d5ed89d1adffa0a" Oct 30 18:21:38.199329 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:21:38.199313 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6afbbbbda0c22724bb992044b34f8f4c8a558230f5a5ba229d5ed89d1adffa0a\": container with ID starting with 6afbbbbda0c22724bb992044b34f8f4c8a558230f5a5ba229d5ed89d1adffa0a not found: ID does not exist" containerID="6afbbbbda0c22724bb992044b34f8f4c8a558230f5a5ba229d5ed89d1adffa0a" Oct 30 18:21:38.199370 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:38.199340 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6afbbbbda0c22724bb992044b34f8f4c8a558230f5a5ba229d5ed89d1adffa0a"} err="failed to get container status \"6afbbbbda0c22724bb992044b34f8f4c8a558230f5a5ba229d5ed89d1adffa0a\": rpc error: code = NotFound desc = could not find container \"6afbbbbda0c22724bb992044b34f8f4c8a558230f5a5ba229d5ed89d1adffa0a\": container with ID starting with 6afbbbbda0c22724bb992044b34f8f4c8a558230f5a5ba229d5ed89d1adffa0a not found: ID does not exist" Oct 30 18:21:38.199370 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:38.199356 4603 scope.go:122] "RemoveContainer" containerID="c7b288f582bf7a4698353f293951d0b773c2be98ba32cb1f8abb2aa0de178bc3" Oct 30 18:21:38.199592 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:21:38.199576 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7b288f582bf7a4698353f293951d0b773c2be98ba32cb1f8abb2aa0de178bc3\": container with ID starting with c7b288f582bf7a4698353f293951d0b773c2be98ba32cb1f8abb2aa0de178bc3 not found: ID does not exist" containerID="c7b288f582bf7a4698353f293951d0b773c2be98ba32cb1f8abb2aa0de178bc3" Oct 30 18:21:38.199637 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:38.199600 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7b288f582bf7a4698353f293951d0b773c2be98ba32cb1f8abb2aa0de178bc3"} err="failed to get container status \"c7b288f582bf7a4698353f293951d0b773c2be98ba32cb1f8abb2aa0de178bc3\": rpc error: code = NotFound desc = could not find container \"c7b288f582bf7a4698353f293951d0b773c2be98ba32cb1f8abb2aa0de178bc3\": container with ID starting with c7b288f582bf7a4698353f293951d0b773c2be98ba32cb1f8abb2aa0de178bc3 not found: ID does not exist" Oct 30 18:21:38.232033 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:38.232011 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl"] Oct 30 18:21:38.243968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:38.243949 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-853d3a-openshift-default-df76d986c-g6mkl"] Oct 30 18:21:38.550300 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:38.550271 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:39.410919 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:39.410851 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7486973-71c0-4102-9f74-c814bf601ec5" path="/var/lib/kubelet/pods/f7486973-71c0-4102-9f74-c814bf601ec5/volumes" Oct 30 18:21:40.135074 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:40.135043 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:41.790612 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:41.790577 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:43.929861 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:43.929828 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:45.587260 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:45.587226 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:46.989980 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:46.989939 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:47.044824 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.044786 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7"] Oct 30 18:21:47.045052 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.045029 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7" podUID="88e882e9-9a9c-4371-a61e-e5b8e707b2ac" containerName="payload-processing" containerID="cri-o://6d33c45dc536ec1fa5c29f9990d1e915a395627922662dce70d94530b80d6994" gracePeriod=30 Oct 30 18:21:47.064634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.064609 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr"] Oct 30 18:21:47.064837 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.064816 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr" podUID="9b5e319e-7f09-43f1-8847-6c6a7cc73ade" containerName="payload-pre-processing" containerID="cri-o://58d4d941dc24ea86bf226f83075df6bc6da6bdc9b50d885b0e94189444334298" gracePeriod=30 Oct 30 18:21:47.157905 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.157872 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:47.271524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.270900 4603 generic.go:356] "Generic (PLEG): container finished" podID="88e882e9-9a9c-4371-a61e-e5b8e707b2ac" containerID="6d33c45dc536ec1fa5c29f9990d1e915a395627922662dce70d94530b80d6994" exitCode=0 Oct 30 18:21:47.271524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.271038 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7" event={"ID":"88e882e9-9a9c-4371-a61e-e5b8e707b2ac","Type":"ContainerDied","Data":"6d33c45dc536ec1fa5c29f9990d1e915a395627922662dce70d94530b80d6994"} Oct 30 18:21:47.272906 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.272838 4603 generic.go:356] "Generic (PLEG): container finished" podID="9b5e319e-7f09-43f1-8847-6c6a7cc73ade" containerID="58d4d941dc24ea86bf226f83075df6bc6da6bdc9b50d885b0e94189444334298" exitCode=0 Oct 30 18:21:47.273029 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.272969 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr" event={"ID":"9b5e319e-7f09-43f1-8847-6c6a7cc73ade","Type":"ContainerDied","Data":"58d4d941dc24ea86bf226f83075df6bc6da6bdc9b50d885b0e94189444334298"} Oct 30 18:21:47.313133 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.313061 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:47.370419 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.370399 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7" Oct 30 18:21:47.414967 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.414938 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr" Oct 30 18:21:47.542577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.538682 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5e319e-7f09-43f1-8847-6c6a7cc73ade-kube-api-access-ddxg8\" (UniqueName: \"kubernetes.io/projected/9b5e319e-7f09-43f1-8847-6c6a7cc73ade-kube-api-access-ddxg8\") pod \"9b5e319e-7f09-43f1-8847-6c6a7cc73ade\" (UID: \"9b5e319e-7f09-43f1-8847-6c6a7cc73ade\") " Oct 30 18:21:47.542577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.538914 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e882e9-9a9c-4371-a61e-e5b8e707b2ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e882e9-9a9c-4371-a61e-e5b8e707b2ac-plugins-config-volume\") pod \"88e882e9-9a9c-4371-a61e-e5b8e707b2ac\" (UID: \"88e882e9-9a9c-4371-a61e-e5b8e707b2ac\") " Oct 30 18:21:47.542577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.539093 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b5e319e-7f09-43f1-8847-6c6a7cc73ade-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5e319e-7f09-43f1-8847-6c6a7cc73ade-plugins-config-volume\") pod \"9b5e319e-7f09-43f1-8847-6c6a7cc73ade\" (UID: \"9b5e319e-7f09-43f1-8847-6c6a7cc73ade\") " Oct 30 18:21:47.542577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.539131 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e882e9-9a9c-4371-a61e-e5b8e707b2ac-kube-api-access-qgzgq\" (UniqueName: \"kubernetes.io/projected/88e882e9-9a9c-4371-a61e-e5b8e707b2ac-kube-api-access-qgzgq\") pod \"88e882e9-9a9c-4371-a61e-e5b8e707b2ac\" (UID: \"88e882e9-9a9c-4371-a61e-e5b8e707b2ac\") " Oct 30 18:21:47.542577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.542233 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e882e9-9a9c-4371-a61e-e5b8e707b2ac-plugins-config-volume" pod "88e882e9-9a9c-4371-a61e-e5b8e707b2ac" (UID: "88e882e9-9a9c-4371-a61e-e5b8e707b2ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:47.572393 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.560099 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:47.709548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.699016 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e882e9-9a9c-4371-a61e-e5b8e707b2ac-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:47.709548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.703940 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5e319e-7f09-43f1-8847-6c6a7cc73ade-plugins-config-volume" pod "9b5e319e-7f09-43f1-8847-6c6a7cc73ade" (UID: "9b5e319e-7f09-43f1-8847-6c6a7cc73ade"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:47.797576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.796397 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e882e9-9a9c-4371-a61e-e5b8e707b2ac-kube-api-access-qgzgq" pod "88e882e9-9a9c-4371-a61e-e5b8e707b2ac" (UID: "88e882e9-9a9c-4371-a61e-e5b8e707b2ac"). InnerVolumeSpecName "kube-api-access-qgzgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:47.797576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.796565 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5e319e-7f09-43f1-8847-6c6a7cc73ade-kube-api-access-ddxg8" pod "9b5e319e-7f09-43f1-8847-6c6a7cc73ade" (UID: "9b5e319e-7f09-43f1-8847-6c6a7cc73ade"). InnerVolumeSpecName "kube-api-access-ddxg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:47.804524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.803818 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddxg8\" (UniqueName: \"kubernetes.io/projected/9b5e319e-7f09-43f1-8847-6c6a7cc73ade-kube-api-access-ddxg8\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:47.804524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.803852 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5e319e-7f09-43f1-8847-6c6a7cc73ade-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:47.804524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.803868 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgzgq\" (UniqueName: \"kubernetes.io/projected/88e882e9-9a9c-4371-a61e-e5b8e707b2ac-kube-api-access-qgzgq\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:21:47.815456 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.815257 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:47.962829 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:47.962792 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:48.240753 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:48.240724 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:48.280680 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:48.280654 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7" Oct 30 18:21:48.280680 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:48.280665 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7" event={"ID":"88e882e9-9a9c-4371-a61e-e5b8e707b2ac","Type":"ContainerDied","Data":"949109d749b2b34f75ffe25305347d424521a1ddd958df52bdb09ab1e9527b47"} Oct 30 18:21:48.280879 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:48.280735 4603 scope.go:122] "RemoveContainer" containerID="6d33c45dc536ec1fa5c29f9990d1e915a395627922662dce70d94530b80d6994" Oct 30 18:21:48.282253 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:48.282222 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr" Oct 30 18:21:48.282394 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:48.282259 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr" event={"ID":"9b5e319e-7f09-43f1-8847-6c6a7cc73ade","Type":"ContainerDied","Data":"3f4c453b8c813b6b2fbc974214ae26b1a939ef060081ee9c7ea707f15bff4f15"} Oct 30 18:21:48.293759 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:48.292348 4603 scope.go:122] "RemoveContainer" containerID="58d4d941dc24ea86bf226f83075df6bc6da6bdc9b50d885b0e94189444334298" Oct 30 18:21:48.333174 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:48.333116 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7"] Oct 30 18:21:48.344993 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:48.344968 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c33ee4-66d46d6784-rqlx7"] Oct 30 18:21:48.378540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:48.378478 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr"] Oct 30 18:21:48.381899 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:48.381881 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c33ee4-5c485df94c-2bjtr"] Oct 30 18:21:49.469526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:49.464272 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:49.497265 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:49.494428 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e882e9-9a9c-4371-a61e-e5b8e707b2ac" path="/var/lib/kubelet/pods/88e882e9-9a9c-4371-a61e-e5b8e707b2ac/volumes" Oct 30 18:21:49.497265 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:49.495081 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b5e319e-7f09-43f1-8847-6c6a7cc73ade" path="/var/lib/kubelet/pods/9b5e319e-7f09-43f1-8847-6c6a7cc73ade/volumes" Oct 30 18:21:51.045821 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:51.045781 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:51.931427 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:21:51.931391 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:21:52.728799 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:52.728765 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:54.447055 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:54.447020 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:54.604195 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:54.604161 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:54.717316 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:54.717236 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl"] Oct 30 18:21:54.717490 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:54.717469 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" podUID="d036618a-de13-4ab8-affb-7fc490084530" containerName="istio-proxy" containerID="cri-o://dc7473b163dd47329282d0a44e6810910153f5d39f2bbaeabcd54f56dc27ff7d" gracePeriod=30 Oct 30 18:21:54.791903 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:54.791869 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:55.029387 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:55.029354 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:55.173837 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:55.173798 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:56.117529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:56.115061 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:21:56.800115 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:56.800084 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:57.448187 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:21:57.448153 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:00.235802 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:00.235655 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:00.368536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:00.325607 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Oct 30 18:22:00.368536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:00.326231 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:22:00.417555 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:00.406662 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:22:00.538913 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:00.538857 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:00.588628 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:00.588579 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:22:00.604675 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:00.604636 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:00.618945 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:00.613052 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:00.632014 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:00.631982 4603 generic.go:356] "Generic (PLEG): container finished" podID="d036618a-de13-4ab8-affb-7fc490084530" containerID="dc7473b163dd47329282d0a44e6810910153f5d39f2bbaeabcd54f56dc27ff7d" exitCode=0 Oct 30 18:22:00.632323 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:00.632306 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" event={"ID":"d036618a-de13-4ab8-affb-7fc490084530","Type":"ContainerDied","Data":"dc7473b163dd47329282d0a44e6810910153f5d39f2bbaeabcd54f56dc27ff7d"} Oct 30 18:22:00.632453 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:00.632442 4603 scope.go:122] "RemoveContainer" containerID="9936aa0093b73d9f0acdb8ae8c2a7ee8b9f9bf2b7b35147dcc208186c9b3bb01" Oct 30 18:22:00.857582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:00.824196 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:00.954424 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:00.954375 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:01.241271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:01.239635 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:22:01.241271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:01.240088 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:22:01.388723 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:01.388672 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:01.389625 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:01.389600 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:22:01.606719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:01.466582 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:01.606719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:01.467184 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:22:01.606719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:01.589667 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:22:01.606719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:01.590585 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:01.680529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:01.643408 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:22:01.792563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:01.727675 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:22:01.968910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:01.968804 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:02.260583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.259862 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:02.343531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.337653 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:22:02.343531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.337733 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:22:02.343531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.337777 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:22:02.343531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.337811 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:22:02.343531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.337844 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:22:02.343531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.337877 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:22:02.421547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.405888 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:22:02.481755 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.481718 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:22:02.483217 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.483191 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:22:02.581293 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.554152 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-workload-certs\") pod \"d036618a-de13-4ab8-affb-7fc490084530\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " Oct 30 18:22:02.616739 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.599518 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-istio-data\") pod \"d036618a-de13-4ab8-affb-7fc490084530\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " Oct 30 18:22:02.616739 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.599573 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d036618a-de13-4ab8-affb-7fc490084530-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d036618a-de13-4ab8-affb-7fc490084530-istio-podinfo\") pod \"d036618a-de13-4ab8-affb-7fc490084530\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " Oct 30 18:22:02.625533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.624997 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d036618a-de13-4ab8-affb-7fc490084530-istio-token\" (UniqueName: \"kubernetes.io/projected/d036618a-de13-4ab8-affb-7fc490084530-istio-token\") pod \"d036618a-de13-4ab8-affb-7fc490084530\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " Oct 30 18:22:02.634936 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.599348 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:22:02.641535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.568617 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-workload-certs" pod "d036618a-de13-4ab8-affb-7fc490084530" (UID: "d036618a-de13-4ab8-affb-7fc490084530"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:02.641535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.622652 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-istio-data" pod "d036618a-de13-4ab8-affb-7fc490084530" (UID: "d036618a-de13-4ab8-affb-7fc490084530"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:02.641535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.598468 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:02.662558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.662211 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:02.722563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.713592 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-workload-socket\") pod \"d036618a-de13-4ab8-affb-7fc490084530\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " Oct 30 18:22:02.722563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.713690 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-istio-envoy\") pod \"d036618a-de13-4ab8-affb-7fc490084530\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " Oct 30 18:22:02.722563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.713740 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-credential-socket\") pod \"d036618a-de13-4ab8-affb-7fc490084530\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " Oct 30 18:22:02.722563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.713918 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d036618a-de13-4ab8-affb-7fc490084530-kube-api-access-x5rzq\" (UniqueName: \"kubernetes.io/projected/d036618a-de13-4ab8-affb-7fc490084530-kube-api-access-x5rzq\") pod \"d036618a-de13-4ab8-affb-7fc490084530\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " Oct 30 18:22:02.722563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.713966 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d036618a-de13-4ab8-affb-7fc490084530-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d036618a-de13-4ab8-affb-7fc490084530-istiod-ca-cert\") pod \"d036618a-de13-4ab8-affb-7fc490084530\" (UID: \"d036618a-de13-4ab8-affb-7fc490084530\") " Oct 30 18:22:02.722563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.714764 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:02.722563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.714789 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:02.725493 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.724954 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-credential-socket" pod "d036618a-de13-4ab8-affb-7fc490084530" (UID: "d036618a-de13-4ab8-affb-7fc490084530"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:02.725493 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.725415 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-workload-socket" pod "d036618a-de13-4ab8-affb-7fc490084530" (UID: "d036618a-de13-4ab8-affb-7fc490084530"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:02.751720 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.751479 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d036618a-de13-4ab8-affb-7fc490084530-istio-token" pod "d036618a-de13-4ab8-affb-7fc490084530" (UID: "d036618a-de13-4ab8-affb-7fc490084530"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:02.790526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.784707 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d036618a-de13-4ab8-affb-7fc490084530-istiod-ca-cert" pod "d036618a-de13-4ab8-affb-7fc490084530" (UID: "d036618a-de13-4ab8-affb-7fc490084530"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:02.858582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.826190 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d036618a-de13-4ab8-affb-7fc490084530-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:02.858582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.826223 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d036618a-de13-4ab8-affb-7fc490084530-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:02.858582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.826236 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:02.858582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:02.826250 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:03.013531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:03.012561 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-istio-envoy" pod "d036618a-de13-4ab8-affb-7fc490084530" (UID: "d036618a-de13-4ab8-affb-7fc490084530"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:03.124530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:03.116240 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:22:03.137850 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:03.137790 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d036618a-de13-4ab8-affb-7fc490084530-istio-podinfo" pod "d036618a-de13-4ab8-affb-7fc490084530" (UID: "d036618a-de13-4ab8-affb-7fc490084530"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:22:03.472585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:03.469151 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d036618a-de13-4ab8-affb-7fc490084530-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:03.472585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:03.469197 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d036618a-de13-4ab8-affb-7fc490084530-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:03.511531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:03.502655 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:03.511531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:03.506662 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:22:03.637536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:03.602939 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d036618a-de13-4ab8-affb-7fc490084530-kube-api-access-x5rzq" pod "d036618a-de13-4ab8-affb-7fc490084530" (UID: "d036618a-de13-4ab8-affb-7fc490084530"). InnerVolumeSpecName "kube-api-access-x5rzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:04.089530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:03.637978 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.127534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:03.640102 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.140135 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:03.640368 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:22:04.211966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:03.947533 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:22:04.211966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:03.294608 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.211966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:04.125584 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.211966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:04.143592 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:04.211966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:04.202451 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" podUID="b664f4ca-736e-4cb1-991d-985d1e6aff34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.211966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:04.201675 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:04.211966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:04.202573 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" podUID="1d187b51-f5c6-4a63-8300-2f01184a15d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.231717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:04.231672 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:04.245845 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:04.245806 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" podUID="4d6cd466-9f15-476d-9f04-6d03176f025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.271631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:04.245645 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:04.289567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:04.260968 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:04.293530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:04.289818 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" podUID="187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.314530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:04.297765 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" podUID="d4a3adf6-fb23-4334-8bab-f800b4e8e089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.356764 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:04.356666 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:04.357032 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:04.356999 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" podUID="b6e43478-8cc1-4452-8bc8-fed14f127896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.371609 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:04.361755 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5rzq\" (UniqueName: \"kubernetes.io/projected/d036618a-de13-4ab8-affb-7fc490084530-kube-api-access-x5rzq\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:04.531529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:04.527594 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.597869 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:04.597768 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:22:04.641149 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:04.641044 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.765530 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:22:04.744443 4603 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.193s" Oct 30 18:22:04.892829 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:04.892704 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" event={"ID":"d036618a-de13-4ab8-affb-7fc490084530","Type":"ContainerDied","Data":"2c6c5ca6f27fa765ab44370c8dc6b6effe3607df05fe81f6c87ec0de333143f1"} Oct 30 18:22:04.893129 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:04.893108 4603 scope.go:122] "RemoveContainer" containerID="dc7473b163dd47329282d0a44e6810910153f5d39f2bbaeabcd54f56dc27ff7d" Oct 30 18:22:04.894661 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:04.894633 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl" Oct 30 18:22:05.108528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:05.098458 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:05.152290 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:05.144728 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:05.152290 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:05.146293 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:22:05.204531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:05.202641 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:22:05.204531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:05.202848 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:22:05.204531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:05.203326 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:05.204991 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:05.204948 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:05.789537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:05.750281 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:05.789537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:05.752393 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:22:05.789537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:05.752586 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:06.048588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:06.043532 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl"] Oct 30 18:22:06.105214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:06.105180 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c33ee4-openshift-default-859fdb69b5-m5vnl"] Oct 30 18:22:06.105559 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:06.105526 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:06.149032 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:06.148992 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:06.218540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:06.206611 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:22:06.218540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:06.207029 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:22:06.218540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:06.207349 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:22:06.219714 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:06.219679 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:06.779530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:06.753605 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:06.779530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:06.754392 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:07.230025 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:07.229928 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:07.244439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:07.244388 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:22:07.272885 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:07.272837 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:07.290739 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:07.290685 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:22:07.291517 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:07.291452 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:22:07.581527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:07.296030 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:07.739353 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:07.739310 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:22:08.004979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:07.828831 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:08.004979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:07.898917 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:08.078692 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:08.078634 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:22:08.086553 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:08.082932 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:22:08.378807 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:08.378746 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:08.379411 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:08.379361 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" podUID="f7882513-ef6d-4b11-871e-39a29ad022df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:08.382015 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:08.381976 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:22:08.390752 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:08.387091 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:22:08.394732 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:08.387167 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:22:08.397157 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:08.397121 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:08.415993 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:08.387196 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:08.416381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:08.415918 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" podUID="d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:08.739583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:08.739322 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:08.739583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:08.739398 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" podUID="58e2469b-dc0d-40ab-bd74-a47c4a9e438b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:08.772093 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:08.748817 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:08.898532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:08.777077 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:08.898532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:08.805715 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 30 18:22:08.898532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:08.823681 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": context deadline exceeded" Oct 30 18:22:08.913651 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:08.899298 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:08.930531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:08.921050 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:22:09.086563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:09.083615 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:22:09.132122 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:09.108681 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:09.220535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:09.180967 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:09.229543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:09.194421 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d036618a-de13-4ab8-affb-7fc490084530" path="/var/lib/kubelet/pods/d036618a-de13-4ab8-affb-7fc490084530/volumes" Oct 30 18:22:09.229543 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:22:09.226783 4603 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.52s" Oct 30 18:22:09.229543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:09.226866 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf"] Oct 30 18:22:09.229543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:09.226901 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg"] Oct 30 18:22:09.229543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:09.227078 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg" podUID="9ad353f8-72ed-494e-ab3c-429d0169a45f" containerName="payload-pre-processing" containerID="cri-o://7927b15a6433ed272d6308189bdd230b2b52ec78e1bb5e01707afcbadfe450b5" gracePeriod=30 Oct 30 18:22:09.229543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:09.228173 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf" podUID="29c302f2-7a93-46f4-9842-426a6622bfcd" containerName="payload-processing" containerID="cri-o://7ad4e5ea9c987aea995059d605edc08d596bda2d328f79ecd3e7eacdfb65fdc5" gracePeriod=30 Oct 30 18:22:09.398560 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:09.391605 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:09.456876 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:09.430639 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:09.461638 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:09.461589 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:09.534589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:09.509878 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:09.571820 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:09.571776 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:22:09.774005 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:09.770124 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:09.853483 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:22:09.852664 4603 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c302f2_7a93_46f4_9842_426a6622bfcd.slice/crio-conmon-7ad4e5ea9c987aea995059d605edc08d596bda2d328f79ecd3e7eacdfb65fdc5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad353f8_72ed_494e_ab3c_429d0169a45f.slice/crio-conmon-7927b15a6433ed272d6308189bdd230b2b52ec78e1bb5e01707afcbadfe450b5.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:22:09.881707 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:09.881665 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:09.881995 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:09.881965 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" podUID="47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:09.912205 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:09.912106 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:09.981577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:09.933436 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:22:10.007522 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.001877 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:10.086970 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.084590 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:10.276805 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.276776 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:22:10.301060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.301028 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:22:10.305490 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.304900 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:22:10.309600 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.309577 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:22:10.312279 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.312260 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:22:10.313947 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.313930 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:22:10.322767 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.322741 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:22:10.371945 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.371916 4603 generic.go:356] "Generic (PLEG): container finished" podID="29c302f2-7a93-46f4-9842-426a6622bfcd" containerID="7ad4e5ea9c987aea995059d605edc08d596bda2d328f79ecd3e7eacdfb65fdc5" exitCode=0 Oct 30 18:22:10.372252 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.372214 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf" event={"ID":"29c302f2-7a93-46f4-9842-426a6622bfcd","Type":"ContainerDied","Data":"7ad4e5ea9c987aea995059d605edc08d596bda2d328f79ecd3e7eacdfb65fdc5"} Oct 30 18:22:10.382952 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.378638 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:22:10.382952 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.381369 4603 generic.go:356] "Generic (PLEG): container finished" podID="9ad353f8-72ed-494e-ab3c-429d0169a45f" containerID="7927b15a6433ed272d6308189bdd230b2b52ec78e1bb5e01707afcbadfe450b5" exitCode=0 Oct 30 18:22:10.382952 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.381457 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg" event={"ID":"9ad353f8-72ed-494e-ab3c-429d0169a45f","Type":"ContainerDied","Data":"7927b15a6433ed272d6308189bdd230b2b52ec78e1bb5e01707afcbadfe450b5"} Oct 30 18:22:10.389599 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.389573 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf" Oct 30 18:22:10.428539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.428402 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29c302f2-7a93-46f4-9842-426a6622bfcd-kube-api-access-x7ch4\" (UniqueName: \"kubernetes.io/projected/29c302f2-7a93-46f4-9842-426a6622bfcd-kube-api-access-x7ch4\") pod \"29c302f2-7a93-46f4-9842-426a6622bfcd\" (UID: \"29c302f2-7a93-46f4-9842-426a6622bfcd\") " Oct 30 18:22:10.429040 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.428798 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29c302f2-7a93-46f4-9842-426a6622bfcd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c302f2-7a93-46f4-9842-426a6622bfcd-plugins-config-volume\") pod \"29c302f2-7a93-46f4-9842-426a6622bfcd\" (UID: \"29c302f2-7a93-46f4-9842-426a6622bfcd\") " Oct 30 18:22:10.429483 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.429457 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c302f2-7a93-46f4-9842-426a6622bfcd-plugins-config-volume" pod "29c302f2-7a93-46f4-9842-426a6622bfcd" (UID: "29c302f2-7a93-46f4-9842-426a6622bfcd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:10.430185 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.430165 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c302f2-7a93-46f4-9842-426a6622bfcd-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:10.436031 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.436003 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c302f2-7a93-46f4-9842-426a6622bfcd-kube-api-access-x7ch4" pod "29c302f2-7a93-46f4-9842-426a6622bfcd" (UID: "29c302f2-7a93-46f4-9842-426a6622bfcd"). InnerVolumeSpecName "kube-api-access-x7ch4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:10.482468 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.482439 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg" Oct 30 18:22:10.531002 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.530914 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ad353f8-72ed-494e-ab3c-429d0169a45f-kube-api-access-k2m6m\" (UniqueName: \"kubernetes.io/projected/9ad353f8-72ed-494e-ab3c-429d0169a45f-kube-api-access-k2m6m\") pod \"9ad353f8-72ed-494e-ab3c-429d0169a45f\" (UID: \"9ad353f8-72ed-494e-ab3c-429d0169a45f\") " Oct 30 18:22:10.531162 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.531048 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ad353f8-72ed-494e-ab3c-429d0169a45f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad353f8-72ed-494e-ab3c-429d0169a45f-plugins-config-volume\") pod \"9ad353f8-72ed-494e-ab3c-429d0169a45f\" (UID: \"9ad353f8-72ed-494e-ab3c-429d0169a45f\") " Oct 30 18:22:10.531455 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.531424 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad353f8-72ed-494e-ab3c-429d0169a45f-plugins-config-volume" pod "9ad353f8-72ed-494e-ab3c-429d0169a45f" (UID: "9ad353f8-72ed-494e-ab3c-429d0169a45f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:10.531758 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.531738 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad353f8-72ed-494e-ab3c-429d0169a45f-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:10.531758 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.531758 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7ch4\" (UniqueName: \"kubernetes.io/projected/29c302f2-7a93-46f4-9842-426a6622bfcd-kube-api-access-x7ch4\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:10.533857 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.533825 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad353f8-72ed-494e-ab3c-429d0169a45f-kube-api-access-k2m6m" pod "9ad353f8-72ed-494e-ab3c-429d0169a45f" (UID: "9ad353f8-72ed-494e-ab3c-429d0169a45f"). InnerVolumeSpecName "kube-api-access-k2m6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:10.633485 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:10.633450 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2m6m\" (UniqueName: \"kubernetes.io/projected/9ad353f8-72ed-494e-ab3c-429d0169a45f-kube-api-access-k2m6m\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:11.389171 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:11.389139 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf" Oct 30 18:22:11.389171 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:11.389160 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf" event={"ID":"29c302f2-7a93-46f4-9842-426a6622bfcd","Type":"ContainerDied","Data":"982f59e9d8829c5b563bcdcf56488e3c3591f8331f1b8073a7572611f6c427c1"} Oct 30 18:22:11.389434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:11.389222 4603 scope.go:122] "RemoveContainer" containerID="7ad4e5ea9c987aea995059d605edc08d596bda2d328f79ecd3e7eacdfb65fdc5" Oct 30 18:22:11.390797 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:11.390771 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg" Oct 30 18:22:11.390904 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:11.390803 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg" event={"ID":"9ad353f8-72ed-494e-ab3c-429d0169a45f","Type":"ContainerDied","Data":"b0928025452fe6612c4715fc4ead80497baa2fd6dad6c90ce55d4b329c11758e"} Oct 30 18:22:11.400027 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:11.400010 4603 scope.go:122] "RemoveContainer" containerID="7927b15a6433ed272d6308189bdd230b2b52ec78e1bb5e01707afcbadfe450b5" Oct 30 18:22:11.440583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:11.440536 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg"] Oct 30 18:22:11.455966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:11.455939 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-755bc9b467-jdqjg"] Oct 30 18:22:11.478584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:11.478559 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf"] Oct 30 18:22:11.484599 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:11.484577 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-189cda-554fd54cb4-ndrgf"] Oct 30 18:22:12.141210 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:12.141171 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:12.387388 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:12.387359 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:13.295842 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:13.295802 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:13.409968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:13.409938 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29c302f2-7a93-46f4-9842-426a6622bfcd" path="/var/lib/kubelet/pods/29c302f2-7a93-46f4-9842-426a6622bfcd/volumes" Oct 30 18:22:13.410374 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:13.410360 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ad353f8-72ed-494e-ab3c-429d0169a45f" path="/var/lib/kubelet/pods/9ad353f8-72ed-494e-ab3c-429d0169a45f/volumes" Oct 30 18:22:14.018115 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:14.018059 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:15.831867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:15.831827 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:15.970885 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:15.970848 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:16.149241 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:16.149152 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6"] Oct 30 18:22:16.149432 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:16.149387 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" podUID="4a7164ea-a62e-4d5d-b26d-adcae50ff87c" containerName="istio-proxy" containerID="cri-o://8d437eb4d0f5bbe38d07e3d7af11ba32f57f3aea1dded6f4ed090e7db5f55cd5" gracePeriod=30 Oct 30 18:22:16.223023 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:16.222993 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:16.391974 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:16.391941 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:17.813846 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:17.813804 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:17.962295 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:17.962262 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:18.218626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.218536 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q"] Oct 30 18:22:18.219421 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219401 4603 cpu_manager.go:414] "containerMap: removing container" podUID="29c302f2-7a93-46f4-9842-426a6622bfcd" containerName="payload-processing" Oct 30 18:22:18.219421 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219420 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c302f2-7a93-46f4-9842-426a6622bfcd" containerName="payload-processing" Oct 30 18:22:18.219647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219437 4603 cpu_manager.go:414] "containerMap: removing container" podUID="18da6086-657d-4917-9fcb-c8de74e7f7b4" containerName="istio-proxy" Oct 30 18:22:18.219647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219445 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18da6086-657d-4917-9fcb-c8de74e7f7b4" containerName="istio-proxy" Oct 30 18:22:18.219647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219458 4603 cpu_manager.go:414] "containerMap: removing container" podUID="6fa3594b-6dab-4f71-a8c1-183151163cfb" containerName="payload-processing" Oct 30 18:22:18.219647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219467 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa3594b-6dab-4f71-a8c1-183151163cfb" containerName="payload-processing" Oct 30 18:22:18.219647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219482 4603 cpu_manager.go:414] "containerMap: removing container" podUID="18da6086-657d-4917-9fcb-c8de74e7f7b4" containerName="istio-proxy" Oct 30 18:22:18.219647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219490 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18da6086-657d-4917-9fcb-c8de74e7f7b4" containerName="istio-proxy" Oct 30 18:22:18.219647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219530 4603 cpu_manager.go:414] "containerMap: removing container" podUID="9ad353f8-72ed-494e-ab3c-429d0169a45f" containerName="payload-pre-processing" Oct 30 18:22:18.219647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219538 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ad353f8-72ed-494e-ab3c-429d0169a45f" containerName="payload-pre-processing" Oct 30 18:22:18.219647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219561 4603 cpu_manager.go:414] "containerMap: removing container" podUID="f7486973-71c0-4102-9f74-c814bf601ec5" containerName="istio-proxy" Oct 30 18:22:18.219647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219568 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7486973-71c0-4102-9f74-c814bf601ec5" containerName="istio-proxy" Oct 30 18:22:18.219647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219593 4603 cpu_manager.go:414] "containerMap: removing container" podUID="9b5e319e-7f09-43f1-8847-6c6a7cc73ade" containerName="payload-pre-processing" Oct 30 18:22:18.219647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219600 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5e319e-7f09-43f1-8847-6c6a7cc73ade" containerName="payload-pre-processing" Oct 30 18:22:18.219647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219617 4603 cpu_manager.go:414] "containerMap: removing container" podUID="ea551dc1-ef6e-4a74-a118-bef1b0f70a57" containerName="payload-pre-processing" Oct 30 18:22:18.219647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219623 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea551dc1-ef6e-4a74-a118-bef1b0f70a57" containerName="payload-pre-processing" Oct 30 18:22:18.219647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219639 4603 cpu_manager.go:414] "containerMap: removing container" podUID="d036618a-de13-4ab8-affb-7fc490084530" containerName="istio-proxy" Oct 30 18:22:18.219647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219643 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d036618a-de13-4ab8-affb-7fc490084530" containerName="istio-proxy" Oct 30 18:22:18.219647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219654 4603 cpu_manager.go:414] "containerMap: removing container" podUID="d036618a-de13-4ab8-affb-7fc490084530" containerName="istio-proxy" Oct 30 18:22:18.219647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219659 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d036618a-de13-4ab8-affb-7fc490084530" containerName="istio-proxy" Oct 30 18:22:18.220571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219666 4603 cpu_manager.go:414] "containerMap: removing container" podUID="88e882e9-9a9c-4371-a61e-e5b8e707b2ac" containerName="payload-processing" Oct 30 18:22:18.220571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219670 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e882e9-9a9c-4371-a61e-e5b8e707b2ac" containerName="payload-processing" Oct 30 18:22:18.220571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219816 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7486973-71c0-4102-9f74-c814bf601ec5" containerName="istio-proxy" Oct 30 18:22:18.220571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219836 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7486973-71c0-4102-9f74-c814bf601ec5" containerName="istio-proxy" Oct 30 18:22:18.220571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219846 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5e319e-7f09-43f1-8847-6c6a7cc73ade" containerName="payload-pre-processing" Oct 30 18:22:18.220571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219857 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea551dc1-ef6e-4a74-a118-bef1b0f70a57" containerName="payload-pre-processing" Oct 30 18:22:18.220571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219868 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="d036618a-de13-4ab8-affb-7fc490084530" containerName="istio-proxy" Oct 30 18:22:18.220571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219877 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="d036618a-de13-4ab8-affb-7fc490084530" containerName="istio-proxy" Oct 30 18:22:18.220571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219885 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e882e9-9a9c-4371-a61e-e5b8e707b2ac" containerName="payload-processing" Oct 30 18:22:18.220571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219892 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="29c302f2-7a93-46f4-9842-426a6622bfcd" containerName="payload-processing" Oct 30 18:22:18.220571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219897 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="18da6086-657d-4917-9fcb-c8de74e7f7b4" containerName="istio-proxy" Oct 30 18:22:18.220571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219903 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa3594b-6dab-4f71-a8c1-183151163cfb" containerName="payload-processing" Oct 30 18:22:18.220571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219909 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="18da6086-657d-4917-9fcb-c8de74e7f7b4" containerName="istio-proxy" Oct 30 18:22:18.220571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.219917 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ad353f8-72ed-494e-ab3c-429d0169a45f" containerName="payload-pre-processing" Oct 30 18:22:18.224128 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.224101 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.243437 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.243408 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q"] Oct 30 18:22:18.250892 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.250872 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-d42172-openshift-default-dockercfg-77zmt\"" Oct 30 18:22:18.292248 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.292223 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:18.402529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.396806 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-token\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.402529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.396890 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-credential-socket\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.402529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.396965 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-data\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.402529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.397011 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-workload-certs\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.402529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.397043 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istiod-ca-cert\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.402529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.397168 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-workload-socket\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.402529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.397208 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zxcl\" (UniqueName: \"kubernetes.io/projected/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-kube-api-access-8zxcl\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.402529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.397304 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-envoy\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.402529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.397413 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-podinfo\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.515595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.514484 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-data\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.515595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.514563 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-workload-certs\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.515595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.514596 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istiod-ca-cert\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.515595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.514694 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-workload-socket\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.515595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.514719 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zxcl\" (UniqueName: \"kubernetes.io/projected/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-kube-api-access-8zxcl\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.515595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.514794 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-envoy\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.515595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.514909 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-podinfo\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.515595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.515040 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-token\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.515595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.515104 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-credential-socket\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.516138 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.515839 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-data\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.516138 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.516086 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-workload-certs\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.530529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.517835 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-credential-socket\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.530529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.518163 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-workload-socket\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.530529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.522966 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istiod-ca-cert\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.575633 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.570460 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-podinfo\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.598555 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.597816 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-envoy\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.700683 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.699823 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-token\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.700683 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.700336 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zxcl\" (UniqueName: \"kubernetes.io/projected/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-kube-api-access-8zxcl\") pod \"e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:18.825523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.822088 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:18.854516 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:18.850958 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:19.233543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:19.233517 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q"] Oct 30 18:22:19.234372 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:22:19.234346 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c7cfe9_00f8_4962_a68f_ec84a2ae84d7.slice/crio-f53a02072773cfea7eda4aee8c81595211e40f416bb8caa9710f8e532aa3b181 WatchSource:0}: Error finding container f53a02072773cfea7eda4aee8c81595211e40f416bb8caa9710f8e532aa3b181: Status 404 returned error can't find the container with id f53a02072773cfea7eda4aee8c81595211e40f416bb8caa9710f8e532aa3b181 Oct 30 18:22:19.236783 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:19.236751 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:19.236878 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:19.236815 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:19.236878 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:19.236845 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:19.479673 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:19.479631 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" event={"ID":"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7","Type":"ContainerStarted","Data":"f63162a2d6737b4440315e5f1da156b440d6a38db7e14306151fbc7d44df959a"} Oct 30 18:22:19.479876 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:19.479701 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" event={"ID":"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7","Type":"ContainerStarted","Data":"f53a02072773cfea7eda4aee8c81595211e40f416bb8caa9710f8e532aa3b181"} Oct 30 18:22:19.525749 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:19.525703 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" podStartSLOduration=1.525680172 podStartE2EDuration="1.525680172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:19.523129118 +0000 UTC m=+1800.238698992" watchObservedRunningTime="2026-10-30 18:22:19.525680172 +0000 UTC m=+1800.241250046" Oct 30 18:22:19.853603 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:19.853255 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:19.945773 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:19.945737 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d"] Oct 30 18:22:20.239549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:20.239423 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:20.610464 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:20.610440 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 18:22:20.887585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:20.885998 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 18:22:20.909530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:20.900062 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:21.435873 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.435847 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:22:21.443492 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.443467 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-credential-socket\") pod \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " Oct 30 18:22:21.443637 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.443520 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-workload-socket\") pod \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " Oct 30 18:22:21.443705 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.443637 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-podinfo\") pod \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " Oct 30 18:22:21.443705 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.443695 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-kube-api-access-ldjkz\" (UniqueName: \"kubernetes.io/projected/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-kube-api-access-ldjkz\") pod \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " Oct 30 18:22:21.443818 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.443731 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-credential-socket" pod "4a7164ea-a62e-4d5d-b26d-adcae50ff87c" (UID: "4a7164ea-a62e-4d5d-b26d-adcae50ff87c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:21.443818 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.443770 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-workload-socket" pod "4a7164ea-a62e-4d5d-b26d-adcae50ff87c" (UID: "4a7164ea-a62e-4d5d-b26d-adcae50ff87c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:21.443818 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.443788 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-data\") pod \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " Oct 30 18:22:21.443948 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.443824 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-token\" (UniqueName: \"kubernetes.io/projected/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-token\") pod \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " Oct 30 18:22:21.443948 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.443909 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-workload-certs\") pod \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " Oct 30 18:22:21.444059 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.443951 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istiod-ca-cert\") pod \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " Oct 30 18:22:21.444059 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.443991 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-envoy\") pod \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\" (UID: \"4a7164ea-a62e-4d5d-b26d-adcae50ff87c\") " Oct 30 18:22:21.444059 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.444023 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-data" pod "4a7164ea-a62e-4d5d-b26d-adcae50ff87c" (UID: "4a7164ea-a62e-4d5d-b26d-adcae50ff87c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:21.444225 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.444204 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-workload-certs" pod "4a7164ea-a62e-4d5d-b26d-adcae50ff87c" (UID: "4a7164ea-a62e-4d5d-b26d-adcae50ff87c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:21.444319 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.444294 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istiod-ca-cert" pod "4a7164ea-a62e-4d5d-b26d-adcae50ff87c" (UID: "4a7164ea-a62e-4d5d-b26d-adcae50ff87c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:21.444890 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.444868 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.445006 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.444895 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.445006 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.444911 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.445006 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.444925 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.445006 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.444938 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.446097 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.446071 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-podinfo" pod "4a7164ea-a62e-4d5d-b26d-adcae50ff87c" (UID: "4a7164ea-a62e-4d5d-b26d-adcae50ff87c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:22:21.446206 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.446101 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-token" pod "4a7164ea-a62e-4d5d-b26d-adcae50ff87c" (UID: "4a7164ea-a62e-4d5d-b26d-adcae50ff87c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:21.446206 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.446179 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-kube-api-access-ldjkz" pod "4a7164ea-a62e-4d5d-b26d-adcae50ff87c" (UID: "4a7164ea-a62e-4d5d-b26d-adcae50ff87c"). InnerVolumeSpecName "kube-api-access-ldjkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:21.446319 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.446271 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-envoy" pod "4a7164ea-a62e-4d5d-b26d-adcae50ff87c" (UID: "4a7164ea-a62e-4d5d-b26d-adcae50ff87c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:21.496817 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.496783 4603 generic.go:356] "Generic (PLEG): container finished" podID="4a7164ea-a62e-4d5d-b26d-adcae50ff87c" containerID="8d437eb4d0f5bbe38d07e3d7af11ba32f57f3aea1dded6f4ed090e7db5f55cd5" exitCode=0 Oct 30 18:22:21.497006 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.496839 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" Oct 30 18:22:21.497006 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.496876 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" event={"ID":"4a7164ea-a62e-4d5d-b26d-adcae50ff87c","Type":"ContainerDied","Data":"8d437eb4d0f5bbe38d07e3d7af11ba32f57f3aea1dded6f4ed090e7db5f55cd5"} Oct 30 18:22:21.497006 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.496954 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6" event={"ID":"4a7164ea-a62e-4d5d-b26d-adcae50ff87c","Type":"ContainerDied","Data":"9b636f28bfe45421a51cd006e25e6d5f3d742836f3c40257770a1b4bd56533a4"} Oct 30 18:22:21.497006 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.496987 4603 scope.go:122] "RemoveContainer" containerID="8d437eb4d0f5bbe38d07e3d7af11ba32f57f3aea1dded6f4ed090e7db5f55cd5" Oct 30 18:22:21.497317 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.497300 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:21.498934 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.498759 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:22:21.509264 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.509230 4603 scope.go:122] "RemoveContainer" containerID="9ccc70e0484185c438cedbb158cf6f81acc660fc8ef12e5eeb6ec63eea527416" Oct 30 18:22:21.521473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.521453 4603 scope.go:122] "RemoveContainer" containerID="8d437eb4d0f5bbe38d07e3d7af11ba32f57f3aea1dded6f4ed090e7db5f55cd5" Oct 30 18:22:21.521961 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:22:21.521941 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d437eb4d0f5bbe38d07e3d7af11ba32f57f3aea1dded6f4ed090e7db5f55cd5\": container with ID starting with 8d437eb4d0f5bbe38d07e3d7af11ba32f57f3aea1dded6f4ed090e7db5f55cd5 not found: ID does not exist" containerID="8d437eb4d0f5bbe38d07e3d7af11ba32f57f3aea1dded6f4ed090e7db5f55cd5" Oct 30 18:22:21.522072 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.521970 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d437eb4d0f5bbe38d07e3d7af11ba32f57f3aea1dded6f4ed090e7db5f55cd5"} err="failed to get container status \"8d437eb4d0f5bbe38d07e3d7af11ba32f57f3aea1dded6f4ed090e7db5f55cd5\": rpc error: code = NotFound desc = could not find container \"8d437eb4d0f5bbe38d07e3d7af11ba32f57f3aea1dded6f4ed090e7db5f55cd5\": container with ID starting with 8d437eb4d0f5bbe38d07e3d7af11ba32f57f3aea1dded6f4ed090e7db5f55cd5 not found: ID does not exist" Oct 30 18:22:21.522072 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.521987 4603 scope.go:122] "RemoveContainer" containerID="9ccc70e0484185c438cedbb158cf6f81acc660fc8ef12e5eeb6ec63eea527416" Oct 30 18:22:21.522308 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:22:21.522289 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ccc70e0484185c438cedbb158cf6f81acc660fc8ef12e5eeb6ec63eea527416\": container with ID starting with 9ccc70e0484185c438cedbb158cf6f81acc660fc8ef12e5eeb6ec63eea527416 not found: ID does not exist" containerID="9ccc70e0484185c438cedbb158cf6f81acc660fc8ef12e5eeb6ec63eea527416" Oct 30 18:22:21.522371 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.522326 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ccc70e0484185c438cedbb158cf6f81acc660fc8ef12e5eeb6ec63eea527416"} err="failed to get container status \"9ccc70e0484185c438cedbb158cf6f81acc660fc8ef12e5eeb6ec63eea527416\": rpc error: code = NotFound desc = could not find container \"9ccc70e0484185c438cedbb158cf6f81acc660fc8ef12e5eeb6ec63eea527416\": container with ID starting with 9ccc70e0484185c438cedbb158cf6f81acc660fc8ef12e5eeb6ec63eea527416 not found: ID does not exist" Oct 30 18:22:21.546837 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.546764 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldjkz\" (UniqueName: \"kubernetes.io/projected/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-kube-api-access-ldjkz\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.546837 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.546794 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.546837 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.546812 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.547212 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.546841 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a7164ea-a62e-4d5d-b26d-adcae50ff87c-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.552232 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.552211 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6"] Oct 30 18:22:21.557610 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.557585 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-29gn6"] Oct 30 18:22:21.930936 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:22:21.930899 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:22:21.990780 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:21.990751 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:23.410454 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:23.410413 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a7164ea-a62e-4d5d-b26d-adcae50ff87c" path="/var/lib/kubelet/pods/4a7164ea-a62e-4d5d-b26d-adcae50ff87c/volumes" Oct 30 18:22:24.128230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:24.128167 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:24.946988 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:24.946948 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d" podUID="f77e84fd-cfe3-47be-9a3a-58759d5a9fc8" containerName="payload-processing" containerID="cri-o://b7c3e014b353eb0e0602ec884ea5a3322efc232b775ba721b49cd6c79f32da3d" gracePeriod=25 Oct 30 18:22:25.227726 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:25.227704 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d" Oct 30 18:22:25.411487 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:25.411439 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f77e84fd-cfe3-47be-9a3a-58759d5a9fc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f77e84fd-cfe3-47be-9a3a-58759d5a9fc8-plugins-config-volume\") pod \"f77e84fd-cfe3-47be-9a3a-58759d5a9fc8\" (UID: \"f77e84fd-cfe3-47be-9a3a-58759d5a9fc8\") " Oct 30 18:22:25.411687 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:25.411620 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f77e84fd-cfe3-47be-9a3a-58759d5a9fc8-kube-api-access-sb7kf\" (UniqueName: \"kubernetes.io/projected/f77e84fd-cfe3-47be-9a3a-58759d5a9fc8-kube-api-access-sb7kf\") pod \"f77e84fd-cfe3-47be-9a3a-58759d5a9fc8\" (UID: \"f77e84fd-cfe3-47be-9a3a-58759d5a9fc8\") " Oct 30 18:22:25.411853 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:25.411830 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f77e84fd-cfe3-47be-9a3a-58759d5a9fc8-plugins-config-volume" pod "f77e84fd-cfe3-47be-9a3a-58759d5a9fc8" (UID: "f77e84fd-cfe3-47be-9a3a-58759d5a9fc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:25.412219 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:25.412201 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f77e84fd-cfe3-47be-9a3a-58759d5a9fc8-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:25.413857 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:25.413832 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f77e84fd-cfe3-47be-9a3a-58759d5a9fc8-kube-api-access-sb7kf" pod "f77e84fd-cfe3-47be-9a3a-58759d5a9fc8" (UID: "f77e84fd-cfe3-47be-9a3a-58759d5a9fc8"). InnerVolumeSpecName "kube-api-access-sb7kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:25.513913 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:25.513848 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb7kf\" (UniqueName: \"kubernetes.io/projected/f77e84fd-cfe3-47be-9a3a-58759d5a9fc8-kube-api-access-sb7kf\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:25.523582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:25.523551 4603 generic.go:356] "Generic (PLEG): container finished" podID="f77e84fd-cfe3-47be-9a3a-58759d5a9fc8" containerID="b7c3e014b353eb0e0602ec884ea5a3322efc232b775ba721b49cd6c79f32da3d" exitCode=0 Oct 30 18:22:25.523710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:25.523611 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d" Oct 30 18:22:25.523710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:25.523623 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d" event={"ID":"f77e84fd-cfe3-47be-9a3a-58759d5a9fc8","Type":"ContainerDied","Data":"b7c3e014b353eb0e0602ec884ea5a3322efc232b775ba721b49cd6c79f32da3d"} Oct 30 18:22:25.523710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:25.523682 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d" event={"ID":"f77e84fd-cfe3-47be-9a3a-58759d5a9fc8","Type":"ContainerDied","Data":"6810da7dbe7592b0a30e24c63976a3f26953d2f5713bd8acdf38709f455c5168"} Oct 30 18:22:25.523710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:25.523706 4603 scope.go:122] "RemoveContainer" containerID="b7c3e014b353eb0e0602ec884ea5a3322efc232b775ba721b49cd6c79f32da3d" Oct 30 18:22:25.534102 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:25.534084 4603 scope.go:122] "RemoveContainer" containerID="b7c3e014b353eb0e0602ec884ea5a3322efc232b775ba721b49cd6c79f32da3d" Oct 30 18:22:25.534357 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:22:25.534335 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7c3e014b353eb0e0602ec884ea5a3322efc232b775ba721b49cd6c79f32da3d\": container with ID starting with b7c3e014b353eb0e0602ec884ea5a3322efc232b775ba721b49cd6c79f32da3d not found: ID does not exist" containerID="b7c3e014b353eb0e0602ec884ea5a3322efc232b775ba721b49cd6c79f32da3d" Oct 30 18:22:25.534409 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:25.534365 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7c3e014b353eb0e0602ec884ea5a3322efc232b775ba721b49cd6c79f32da3d"} err="failed to get container status \"b7c3e014b353eb0e0602ec884ea5a3322efc232b775ba721b49cd6c79f32da3d\": rpc error: code = NotFound desc = could not find container \"b7c3e014b353eb0e0602ec884ea5a3322efc232b775ba721b49cd6c79f32da3d\": container with ID starting with b7c3e014b353eb0e0602ec884ea5a3322efc232b775ba721b49cd6c79f32da3d not found: ID does not exist" Oct 30 18:22:25.574806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:25.574780 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d"] Oct 30 18:22:25.586842 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:25.586820 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a3be60-55c8d87864-g8q8d"] Oct 30 18:22:25.773890 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:25.773857 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:25.871278 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:25.871250 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp"] Oct 30 18:22:25.871471 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:25.871451 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp" podUID="67deb4fb-e2e9-41ea-b2e9-530f568d96ed" containerName="payload-pre-processing" containerID="cri-o://38bd18fe8a7870ee2519aba62bed333f5ff80d8722d1e601d088ff1bff60d287" gracePeriod=30 Oct 30 18:22:25.955274 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:25.955243 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:26.168711 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.168681 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp" Oct 30 18:22:26.204666 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.204632 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:26.331086 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.330998 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67deb4fb-e2e9-41ea-b2e9-530f568d96ed-kube-api-access-k7vt6\" (UniqueName: \"kubernetes.io/projected/67deb4fb-e2e9-41ea-b2e9-530f568d96ed-kube-api-access-k7vt6\") pod \"67deb4fb-e2e9-41ea-b2e9-530f568d96ed\" (UID: \"67deb4fb-e2e9-41ea-b2e9-530f568d96ed\") " Oct 30 18:22:26.331438 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.331417 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67deb4fb-e2e9-41ea-b2e9-530f568d96ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67deb4fb-e2e9-41ea-b2e9-530f568d96ed-plugins-config-volume\") pod \"67deb4fb-e2e9-41ea-b2e9-530f568d96ed\" (UID: \"67deb4fb-e2e9-41ea-b2e9-530f568d96ed\") " Oct 30 18:22:26.331864 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.331817 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67deb4fb-e2e9-41ea-b2e9-530f568d96ed-plugins-config-volume" pod "67deb4fb-e2e9-41ea-b2e9-530f568d96ed" (UID: "67deb4fb-e2e9-41ea-b2e9-530f568d96ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:26.332039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.332020 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67deb4fb-e2e9-41ea-b2e9-530f568d96ed-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:26.333326 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.333302 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67deb4fb-e2e9-41ea-b2e9-530f568d96ed-kube-api-access-k7vt6" pod "67deb4fb-e2e9-41ea-b2e9-530f568d96ed" (UID: "67deb4fb-e2e9-41ea-b2e9-530f568d96ed"). InnerVolumeSpecName "kube-api-access-k7vt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:26.342730 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.342703 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:26.390015 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.389993 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f"] Oct 30 18:22:26.390231 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.390212 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f" podUID="fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba" containerName="payload-processing" containerID="cri-o://1b6db1275eb9474b1eaf26105e24fe04c93c63ea058ed02013fe45b47d4c7d03" gracePeriod=30 Oct 30 18:22:26.434135 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.434103 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7vt6\" (UniqueName: \"kubernetes.io/projected/67deb4fb-e2e9-41ea-b2e9-530f568d96ed-kube-api-access-k7vt6\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:26.491014 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.490961 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:26.532575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.532540 4603 generic.go:356] "Generic (PLEG): container finished" podID="fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba" containerID="1b6db1275eb9474b1eaf26105e24fe04c93c63ea058ed02013fe45b47d4c7d03" exitCode=0 Oct 30 18:22:26.532734 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.532619 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f" event={"ID":"fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba","Type":"ContainerDied","Data":"1b6db1275eb9474b1eaf26105e24fe04c93c63ea058ed02013fe45b47d4c7d03"} Oct 30 18:22:26.535717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.535686 4603 generic.go:356] "Generic (PLEG): container finished" podID="67deb4fb-e2e9-41ea-b2e9-530f568d96ed" containerID="38bd18fe8a7870ee2519aba62bed333f5ff80d8722d1e601d088ff1bff60d287" exitCode=0 Oct 30 18:22:26.535872 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.535796 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp" event={"ID":"67deb4fb-e2e9-41ea-b2e9-530f568d96ed","Type":"ContainerDied","Data":"38bd18fe8a7870ee2519aba62bed333f5ff80d8722d1e601d088ff1bff60d287"} Oct 30 18:22:26.535872 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.535839 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp" event={"ID":"67deb4fb-e2e9-41ea-b2e9-530f568d96ed","Type":"ContainerDied","Data":"4b720ba95a4cf6ebcc9109e4ab6071bac991f447af3a58a6ed2215cc69cf4f50"} Oct 30 18:22:26.535872 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.535867 4603 scope.go:122] "RemoveContainer" containerID="38bd18fe8a7870ee2519aba62bed333f5ff80d8722d1e601d088ff1bff60d287" Oct 30 18:22:26.536059 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.535976 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp" Oct 30 18:22:26.549165 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.549146 4603 scope.go:122] "RemoveContainer" containerID="38bd18fe8a7870ee2519aba62bed333f5ff80d8722d1e601d088ff1bff60d287" Oct 30 18:22:26.549451 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:22:26.549432 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38bd18fe8a7870ee2519aba62bed333f5ff80d8722d1e601d088ff1bff60d287\": container with ID starting with 38bd18fe8a7870ee2519aba62bed333f5ff80d8722d1e601d088ff1bff60d287 not found: ID does not exist" containerID="38bd18fe8a7870ee2519aba62bed333f5ff80d8722d1e601d088ff1bff60d287" Oct 30 18:22:26.549494 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.549461 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38bd18fe8a7870ee2519aba62bed333f5ff80d8722d1e601d088ff1bff60d287"} err="failed to get container status \"38bd18fe8a7870ee2519aba62bed333f5ff80d8722d1e601d088ff1bff60d287\": rpc error: code = NotFound desc = could not find container \"38bd18fe8a7870ee2519aba62bed333f5ff80d8722d1e601d088ff1bff60d287\": container with ID starting with 38bd18fe8a7870ee2519aba62bed333f5ff80d8722d1e601d088ff1bff60d287 not found: ID does not exist" Oct 30 18:22:26.590161 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.588712 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp"] Oct 30 18:22:26.596664 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.596635 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-6bdcf74655-2mmdp"] Oct 30 18:22:26.639535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.639472 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:26.660131 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.660112 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f" Oct 30 18:22:26.788532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.785074 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba-kube-api-access-ppwcm\" (UniqueName: \"kubernetes.io/projected/fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba-kube-api-access-ppwcm\") pod \"fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba\" (UID: \"fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba\") " Oct 30 18:22:26.788532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.785227 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba-plugins-config-volume\") pod \"fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba\" (UID: \"fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba\") " Oct 30 18:22:26.830707 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.824752 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba-plugins-config-volume" pod "fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba" (UID: "fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:26.935746 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.931023 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:26.947530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:26.946677 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba-kube-api-access-ppwcm" pod "fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba" (UID: "fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba"). InnerVolumeSpecName "kube-api-access-ppwcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:27.031529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:27.031430 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:27.041773 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:27.032384 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppwcm\" (UniqueName: \"kubernetes.io/projected/fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba-kube-api-access-ppwcm\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:27.174544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:27.174495 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:27.345770 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:27.345729 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:27.410959 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:27.410931 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67deb4fb-e2e9-41ea-b2e9-530f568d96ed" path="/var/lib/kubelet/pods/67deb4fb-e2e9-41ea-b2e9-530f568d96ed/volumes" Oct 30 18:22:27.411470 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:27.411454 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f77e84fd-cfe3-47be-9a3a-58759d5a9fc8" path="/var/lib/kubelet/pods/f77e84fd-cfe3-47be-9a3a-58759d5a9fc8/volumes" Oct 30 18:22:27.568194 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:27.568158 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f" Oct 30 18:22:27.568194 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:27.568167 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f" event={"ID":"fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba","Type":"ContainerDied","Data":"3cdaeb3c408af06f0cd7280290c0c654d71d3e95f0531b6a0683f1cf2c3f10ce"} Oct 30 18:22:27.568428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:27.568237 4603 scope.go:122] "RemoveContainer" containerID="1b6db1275eb9474b1eaf26105e24fe04c93c63ea058ed02013fe45b47d4c7d03" Oct 30 18:22:27.620047 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:27.619988 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f"] Oct 30 18:22:27.631700 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:27.631676 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-069812-7cd7895d76-jcm7f"] Oct 30 18:22:27.680777 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:27.680751 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:29.410955 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:29.410919 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba" path="/var/lib/kubelet/pods/fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba/volumes" Oct 30 18:22:29.669722 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:29.669639 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:31.359410 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:31.359379 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:32.084778 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:32.084743 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:32.226544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:32.226494 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:33.410309 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:33.410222 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:33.855447 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:33.855401 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:34.000762 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.000729 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:34.367664 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.363918 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85"] Oct 30 18:22:34.382280 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.382246 4603 cpu_manager.go:414] "containerMap: removing container" podUID="4a7164ea-a62e-4d5d-b26d-adcae50ff87c" containerName="istio-proxy" Oct 30 18:22:34.382553 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.382529 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a7164ea-a62e-4d5d-b26d-adcae50ff87c" containerName="istio-proxy" Oct 30 18:22:34.383260 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.383238 4603 cpu_manager.go:414] "containerMap: removing container" podUID="f7486973-71c0-4102-9f74-c814bf601ec5" containerName="istio-proxy" Oct 30 18:22:34.385568 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.385548 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7486973-71c0-4102-9f74-c814bf601ec5" containerName="istio-proxy" Oct 30 18:22:34.385706 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.385693 4603 cpu_manager.go:414] "containerMap: removing container" podUID="4a7164ea-a62e-4d5d-b26d-adcae50ff87c" containerName="istio-proxy" Oct 30 18:22:34.385803 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.385791 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a7164ea-a62e-4d5d-b26d-adcae50ff87c" containerName="istio-proxy" Oct 30 18:22:34.385902 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.385893 4603 cpu_manager.go:414] "containerMap: removing container" podUID="67deb4fb-e2e9-41ea-b2e9-530f568d96ed" containerName="payload-pre-processing" Oct 30 18:22:34.385977 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.385967 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67deb4fb-e2e9-41ea-b2e9-530f568d96ed" containerName="payload-pre-processing" Oct 30 18:22:34.386100 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.386088 4603 cpu_manager.go:414] "containerMap: removing container" podUID="fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba" containerName="payload-processing" Oct 30 18:22:34.386235 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.386222 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba" containerName="payload-processing" Oct 30 18:22:34.386392 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.386382 4603 cpu_manager.go:414] "containerMap: removing container" podUID="f77e84fd-cfe3-47be-9a3a-58759d5a9fc8" containerName="payload-processing" Oct 30 18:22:34.386560 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.386545 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f77e84fd-cfe3-47be-9a3a-58759d5a9fc8" containerName="payload-processing" Oct 30 18:22:34.389429 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.389327 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="f77e84fd-cfe3-47be-9a3a-58759d5a9fc8" containerName="payload-processing" Oct 30 18:22:34.389596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.389577 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a7164ea-a62e-4d5d-b26d-adcae50ff87c" containerName="istio-proxy" Oct 30 18:22:34.389684 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.389611 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a7164ea-a62e-4d5d-b26d-adcae50ff87c" containerName="istio-proxy" Oct 30 18:22:34.389684 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.389637 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="67deb4fb-e2e9-41ea-b2e9-530f568d96ed" containerName="payload-pre-processing" Oct 30 18:22:34.389684 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.389663 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa3d340a-2ea0-4c55-8a5f-ed1b5c8f3bba" containerName="payload-processing" Oct 30 18:22:34.421207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.412759 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:34.423227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.423178 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85"] Oct 30 18:22:34.423625 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.423470 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85" Oct 30 18:22:34.457382 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.457332 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-d42172\"" Oct 30 18:22:34.478894 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.478865 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-d42172-dockercfg-fdp2l\"" Oct 30 18:22:34.487040 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.487005 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85\" (UID: \"fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85" Oct 30 18:22:34.487194 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.487061 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkt6n\" (UniqueName: \"kubernetes.io/projected/fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef-kube-api-access-wkt6n\") pod \"payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85\" (UID: \"fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85" Oct 30 18:22:34.588097 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.588062 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:34.588615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.588581 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85\" (UID: \"fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85" Oct 30 18:22:34.588769 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.588644 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkt6n\" (UniqueName: \"kubernetes.io/projected/fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef-kube-api-access-wkt6n\") pod \"payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85\" (UID: \"fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85" Oct 30 18:22:34.589997 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.589967 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85\" (UID: \"fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85" Oct 30 18:22:34.611677 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.611657 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkt6n\" (UniqueName: \"kubernetes.io/projected/fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef-kube-api-access-wkt6n\") pod \"payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85\" (UID: \"fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85" Oct 30 18:22:34.723283 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.723200 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx"] Oct 30 18:22:34.723552 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.723485 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" podUID="904c45e3-164e-45ee-b877-b8b565fbb7fd" containerName="istio-proxy" containerID="cri-o://c24b5036c142521f0dd0323bb8c916177cfe4f0304d07a6eecbb0055150a4689" gracePeriod=30 Oct 30 18:22:34.783792 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.783765 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85" Oct 30 18:22:34.791830 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:34.791797 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:35.126529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:35.098312 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:35.272000 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:35.267773 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:35.331757 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:35.329423 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85"] Oct 30 18:22:35.353918 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:22:35.353038 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd53dcf5_bf97_41b1_bb6b_96b9bb5383ef.slice/crio-ba2c0d3179a0fc70ea819e058683f3cebeb351b816e135faf8973c6502c33a59 WatchSource:0}: Error finding container ba2c0d3179a0fc70ea819e058683f3cebeb351b816e135faf8973c6502c33a59: Status 404 returned error can't find the container with id ba2c0d3179a0fc70ea819e058683f3cebeb351b816e135faf8973c6502c33a59 Oct 30 18:22:35.354668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:35.354230 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb"] Oct 30 18:22:35.354668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:35.354464 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" podUID="3472e0a8-d76d-45af-a9f4-cae9b46712b6" containerName="istio-proxy" containerID="cri-o://0bccd0424b6f1d37697cbb71196d9656ae3e70d05810e8f2716bc744e3c2db48" gracePeriod=30 Oct 30 18:22:35.436046 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:35.436022 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:35.641569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:35.636488 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85" event={"ID":"fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef","Type":"ContainerStarted","Data":"45ba5ac3525f19ce149c17114894b57d932897a6a433b46ca9256af4c37ad121"} Oct 30 18:22:35.641569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:35.636568 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85" event={"ID":"fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef","Type":"ContainerStarted","Data":"ba2c0d3179a0fc70ea819e058683f3cebeb351b816e135faf8973c6502c33a59"} Oct 30 18:22:35.641569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:35.637792 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85" Oct 30 18:22:35.680713 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:35.680654 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85" podStartSLOduration=1.6806361060000001 podStartE2EDuration="1.680636106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:35.678197403 +0000 UTC m=+1816.393767278" watchObservedRunningTime="2026-10-30 18:22:35.680636106 +0000 UTC m=+1816.396206029" Oct 30 18:22:35.854750 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:35.854721 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:36.348594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.348561 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:36.736475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.736394 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs"] Oct 30 18:22:36.740404 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.740379 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.756712 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.756633 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-ca5f33-openshift-default-dockercfg-svqrr\"" Oct 30 18:22:36.771803 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.771778 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs"] Oct 30 18:22:36.805318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.805286 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-podinfo\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.805529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.805343 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-envoy\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.805529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.805480 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-workload-socket\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.805668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.805581 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-data\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.805668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.805636 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-token\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.805788 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.805685 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-credential-socket\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.805846 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.805790 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf4dp\" (UniqueName: \"kubernetes.io/projected/9d3f60a0-c196-48e7-9765-b1f3d8965367-kube-api-access-lf4dp\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.805977 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.805956 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3f60a0-c196-48e7-9765-b1f3d8965367-istiod-ca-cert\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.806050 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.805988 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-workload-certs\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.831539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.831493 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:36.907173 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.907145 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-podinfo\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.907291 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.907189 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-envoy\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.907291 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.907234 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-workload-socket\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.907291 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.907265 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-data\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.907441 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.907300 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-token\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.907441 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.907341 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-credential-socket\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.907441 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.907390 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf4dp\" (UniqueName: \"kubernetes.io/projected/9d3f60a0-c196-48e7-9765-b1f3d8965367-kube-api-access-lf4dp\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.907622 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.907524 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3f60a0-c196-48e7-9765-b1f3d8965367-istiod-ca-cert\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.907622 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.907546 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-workload-certs\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.907915 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.907885 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-workload-socket\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.908114 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.908085 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-credential-socket\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.908215 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.908155 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-data\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.908287 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.908216 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-workload-certs\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.908287 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.908275 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3f60a0-c196-48e7-9765-b1f3d8965367-istiod-ca-cert\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.909883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.909858 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-podinfo\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.909981 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.909915 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-envoy\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.930324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.930297 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-token\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.930418 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.930360 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf4dp\" (UniqueName: \"kubernetes.io/projected/9d3f60a0-c196-48e7-9765-b1f3d8965367-kube-api-access-lf4dp\") pod \"e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:36.989586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:36.985057 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:37.075557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.060795 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:37.566257 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.565463 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:37.799668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.796065 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:37.799668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.796189 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:37.799668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.796245 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:37.817841 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.817774 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs"] Oct 30 18:22:37.829789 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.829764 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss"] Oct 30 18:22:37.835780 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.835742 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss"] Oct 30 18:22:37.835957 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.835911 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.847293 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.847260 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13daa509-1252-49e8-80ad-caeaedeec185-istio-token\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.847416 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.847315 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-workload-certs\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.847416 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.847353 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-workload-socket\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.847596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.847440 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13daa509-1252-49e8-80ad-caeaedeec185-istiod-ca-cert\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.847596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.847467 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkjkx\" (UniqueName: \"kubernetes.io/projected/13daa509-1252-49e8-80ad-caeaedeec185-kube-api-access-tkjkx\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.847596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.847484 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13daa509-1252-49e8-80ad-caeaedeec185-istio-podinfo\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.847596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.847531 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-istio-envoy\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.847596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.847565 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-istio-data\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.847822 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.847693 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-credential-socket\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.850603 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.850581 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-ef208a-openshift-default-dockercfg-ms84b\"" Oct 30 18:22:37.879468 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.879441 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:37.950497 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.950112 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13daa509-1252-49e8-80ad-caeaedeec185-istio-token\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.950497 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.950224 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-workload-certs\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.950497 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.950313 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-workload-socket\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.950497 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.950382 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13daa509-1252-49e8-80ad-caeaedeec185-istiod-ca-cert\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.950497 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.950426 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkjkx\" (UniqueName: \"kubernetes.io/projected/13daa509-1252-49e8-80ad-caeaedeec185-kube-api-access-tkjkx\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.950497 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.950468 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13daa509-1252-49e8-80ad-caeaedeec185-istio-podinfo\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.951176 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.950531 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-istio-envoy\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.951176 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.950583 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-istio-data\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.951176 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.950668 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-credential-socket\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.951381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.951312 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-credential-socket\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.952080 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.951926 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-workload-certs\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.952212 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.952190 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-workload-socket\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.960526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.956988 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13daa509-1252-49e8-80ad-caeaedeec185-istiod-ca-cert\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.960526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.958341 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-istio-data\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.964567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.963104 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-istio-envoy\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:37.977527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:37.974839 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13daa509-1252-49e8-80ad-caeaedeec185-istio-podinfo\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:38.007528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:38.007184 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13daa509-1252-49e8-80ad-caeaedeec185-istio-token\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:38.110574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:38.106235 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkjkx\" (UniqueName: \"kubernetes.io/projected/13daa509-1252-49e8-80ad-caeaedeec185-kube-api-access-tkjkx\") pod \"e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:38.110574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:38.108859 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:38.167222 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:38.167190 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:38.503200 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:38.503178 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss"] Oct 30 18:22:38.504467 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:22:38.504440 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13daa509_1252_49e8_80ad_caeaedeec185.slice/crio-2afccea54ff7870e1e8d9a0c0eff5ac1a6de2878d8d28d38e69b70e97c9ef247 WatchSource:0}: Error finding container 2afccea54ff7870e1e8d9a0c0eff5ac1a6de2878d8d28d38e69b70e97c9ef247: Status 404 returned error can't find the container with id 2afccea54ff7870e1e8d9a0c0eff5ac1a6de2878d8d28d38e69b70e97c9ef247 Oct 30 18:22:38.507111 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:38.507083 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:38.507193 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:38.507142 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:38.507193 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:38.507172 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:38.835483 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:38.835452 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" event={"ID":"9d3f60a0-c196-48e7-9765-b1f3d8965367","Type":"ContainerStarted","Data":"69dadf26749f14c3df88d4f5b7b25f26514065a46b9b7b9e74baef4efec33e3c"} Oct 30 18:22:38.835921 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:38.835557 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" event={"ID":"9d3f60a0-c196-48e7-9765-b1f3d8965367","Type":"ContainerStarted","Data":"2471aa6c6e5ef6a986317a5ba2578378470b147124ec5f414c5af62b46a4dcec"} Oct 30 18:22:38.837409 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:38.837386 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" event={"ID":"13daa509-1252-49e8-80ad-caeaedeec185","Type":"ContainerStarted","Data":"a7716bf32ed1ffe7b93edb296974cbf967ad8f0772a020f1001ce9320da2e4ce"} Oct 30 18:22:38.837552 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:38.837432 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" event={"ID":"13daa509-1252-49e8-80ad-caeaedeec185","Type":"ContainerStarted","Data":"2afccea54ff7870e1e8d9a0c0eff5ac1a6de2878d8d28d38e69b70e97c9ef247"} Oct 30 18:22:38.881830 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:38.881781 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" podStartSLOduration=2.881768484 podStartE2EDuration="2.881768484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:38.879359919 +0000 UTC m=+1819.594929806" watchObservedRunningTime="2026-10-30 18:22:38.881768484 +0000 UTC m=+1819.597338359" Oct 30 18:22:38.924491 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:38.924438 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" podStartSLOduration=1.924427052 podStartE2EDuration="1.924427052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:38.922984033 +0000 UTC m=+1819.638553931" watchObservedRunningTime="2026-10-30 18:22:38.924427052 +0000 UTC m=+1819.639996925" Oct 30 18:22:39.074530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:39.064867 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:39.160575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:39.158002 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:39.170335 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:39.167883 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:39.185247 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:39.183300 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:39.403742 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:39.403709 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:39.846706 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:39.846674 4603 generic.go:356] "Generic (PLEG): container finished" podID="904c45e3-164e-45ee-b877-b8b565fbb7fd" containerID="c24b5036c142521f0dd0323bb8c916177cfe4f0304d07a6eecbb0055150a4689" exitCode=0 Oct 30 18:22:39.847035 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:39.846736 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" event={"ID":"904c45e3-164e-45ee-b877-b8b565fbb7fd","Type":"ContainerDied","Data":"c24b5036c142521f0dd0323bb8c916177cfe4f0304d07a6eecbb0055150a4689"} Oct 30 18:22:39.847035 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:39.846798 4603 scope.go:122] "RemoveContainer" containerID="017d1e8d557fa8e9be5fb47c79ee543b2df3619eb2969c6a4350433fbba9abfa" Oct 30 18:22:39.847559 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:39.847391 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:39.847559 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:39.847445 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:39.848537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:39.848490 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:22:39.848785 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:39.848770 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:22:40.038282 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.038256 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:22:40.101677 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.101648 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-podinfo\") pod \"904c45e3-164e-45ee-b877-b8b565fbb7fd\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " Oct 30 18:22:40.101885 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.101722 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/904c45e3-164e-45ee-b877-b8b565fbb7fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904c45e3-164e-45ee-b877-b8b565fbb7fd-istiod-ca-cert\") pod \"904c45e3-164e-45ee-b877-b8b565fbb7fd\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " Oct 30 18:22:40.101885 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.101740 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-envoy\") pod \"904c45e3-164e-45ee-b877-b8b565fbb7fd\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " Oct 30 18:22:40.101885 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.101847 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-workload-socket\") pod \"904c45e3-164e-45ee-b877-b8b565fbb7fd\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " Oct 30 18:22:40.102088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.101939 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-data\") pod \"904c45e3-164e-45ee-b877-b8b565fbb7fd\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " Oct 30 18:22:40.102088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.101986 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-token\" (UniqueName: \"kubernetes.io/projected/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-token\") pod \"904c45e3-164e-45ee-b877-b8b565fbb7fd\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " Oct 30 18:22:40.102088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.102024 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-credential-socket\") pod \"904c45e3-164e-45ee-b877-b8b565fbb7fd\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " Oct 30 18:22:40.102088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.102059 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/904c45e3-164e-45ee-b877-b8b565fbb7fd-kube-api-access-vx2r2\" (UniqueName: \"kubernetes.io/projected/904c45e3-164e-45ee-b877-b8b565fbb7fd-kube-api-access-vx2r2\") pod \"904c45e3-164e-45ee-b877-b8b565fbb7fd\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " Oct 30 18:22:40.102088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.102077 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-workload-certs\") pod \"904c45e3-164e-45ee-b877-b8b565fbb7fd\" (UID: \"904c45e3-164e-45ee-b877-b8b565fbb7fd\") " Oct 30 18:22:40.102904 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.102871 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904c45e3-164e-45ee-b877-b8b565fbb7fd-istiod-ca-cert" pod "904c45e3-164e-45ee-b877-b8b565fbb7fd" (UID: "904c45e3-164e-45ee-b877-b8b565fbb7fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:40.103090 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.103064 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-workload-socket" pod "904c45e3-164e-45ee-b877-b8b565fbb7fd" (UID: "904c45e3-164e-45ee-b877-b8b565fbb7fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:40.103389 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.103360 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-credential-socket" pod "904c45e3-164e-45ee-b877-b8b565fbb7fd" (UID: "904c45e3-164e-45ee-b877-b8b565fbb7fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:40.103686 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.103449 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-data" pod "904c45e3-164e-45ee-b877-b8b565fbb7fd" (UID: "904c45e3-164e-45ee-b877-b8b565fbb7fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:40.103686 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.103638 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-workload-certs" pod "904c45e3-164e-45ee-b877-b8b565fbb7fd" (UID: "904c45e3-164e-45ee-b877-b8b565fbb7fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:40.105333 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.105313 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-token" pod "904c45e3-164e-45ee-b877-b8b565fbb7fd" (UID: "904c45e3-164e-45ee-b877-b8b565fbb7fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:40.106411 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.106385 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-podinfo" pod "904c45e3-164e-45ee-b877-b8b565fbb7fd" (UID: "904c45e3-164e-45ee-b877-b8b565fbb7fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:22:40.106879 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.106857 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-envoy" pod "904c45e3-164e-45ee-b877-b8b565fbb7fd" (UID: "904c45e3-164e-45ee-b877-b8b565fbb7fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:40.110882 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.110854 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904c45e3-164e-45ee-b877-b8b565fbb7fd-kube-api-access-vx2r2" pod "904c45e3-164e-45ee-b877-b8b565fbb7fd" (UID: "904c45e3-164e-45ee-b877-b8b565fbb7fd"). InnerVolumeSpecName "kube-api-access-vx2r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:40.207650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.207615 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:40.207650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.207651 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:40.207893 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.207669 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:40.207893 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.207683 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:40.207893 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.207704 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx2r2\" (UniqueName: \"kubernetes.io/projected/904c45e3-164e-45ee-b877-b8b565fbb7fd-kube-api-access-vx2r2\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:40.207893 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.207720 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:40.207893 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.207773 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:40.207893 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.207785 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904c45e3-164e-45ee-b877-b8b565fbb7fd-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:40.207893 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.207804 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904c45e3-164e-45ee-b877-b8b565fbb7fd-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:40.532467 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.532438 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:22:40.612579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.612540 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-data\") pod \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " Oct 30 18:22:40.612762 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.612691 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-workload-socket\") pod \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " Oct 30 18:22:40.612762 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.612737 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-workload-certs\") pod \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " Oct 30 18:22:40.612900 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.612752 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-data" pod "3472e0a8-d76d-45af-a9f4-cae9b46712b6" (UID: "3472e0a8-d76d-45af-a9f4-cae9b46712b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:40.612900 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.612794 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-credential-socket\") pod \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " Oct 30 18:22:40.612900 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.612851 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istiod-ca-cert\") pod \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " Oct 30 18:22:40.613347 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.612923 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3472e0a8-d76d-45af-a9f4-cae9b46712b6-kube-api-access-wq5bv\" (UniqueName: \"kubernetes.io/projected/3472e0a8-d76d-45af-a9f4-cae9b46712b6-kube-api-access-wq5bv\") pod \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " Oct 30 18:22:40.613481 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.612985 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-workload-socket" pod "3472e0a8-d76d-45af-a9f4-cae9b46712b6" (UID: "3472e0a8-d76d-45af-a9f4-cae9b46712b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:40.613481 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.613061 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-workload-certs" pod "3472e0a8-d76d-45af-a9f4-cae9b46712b6" (UID: "3472e0a8-d76d-45af-a9f4-cae9b46712b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:40.613481 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.613103 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-credential-socket" pod "3472e0a8-d76d-45af-a9f4-cae9b46712b6" (UID: "3472e0a8-d76d-45af-a9f4-cae9b46712b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:40.613481 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.613293 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istiod-ca-cert" pod "3472e0a8-d76d-45af-a9f4-cae9b46712b6" (UID: "3472e0a8-d76d-45af-a9f4-cae9b46712b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:40.613753 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.613483 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-token\" (UniqueName: \"kubernetes.io/projected/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-token\") pod \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " Oct 30 18:22:40.613753 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.613583 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-podinfo\") pod \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " Oct 30 18:22:40.613753 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.613636 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-envoy\") pod \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\" (UID: \"3472e0a8-d76d-45af-a9f4-cae9b46712b6\") " Oct 30 18:22:40.614355 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.614327 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:40.614469 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.614359 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:40.614469 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.614374 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:40.614469 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.614389 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:40.614469 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.614402 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:40.616020 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.615990 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3472e0a8-d76d-45af-a9f4-cae9b46712b6-kube-api-access-wq5bv" pod "3472e0a8-d76d-45af-a9f4-cae9b46712b6" (UID: "3472e0a8-d76d-45af-a9f4-cae9b46712b6"). InnerVolumeSpecName "kube-api-access-wq5bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:40.616352 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.616325 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-token" pod "3472e0a8-d76d-45af-a9f4-cae9b46712b6" (UID: "3472e0a8-d76d-45af-a9f4-cae9b46712b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:40.616578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.616557 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-podinfo" pod "3472e0a8-d76d-45af-a9f4-cae9b46712b6" (UID: "3472e0a8-d76d-45af-a9f4-cae9b46712b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:22:40.616847 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.616826 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-envoy" pod "3472e0a8-d76d-45af-a9f4-cae9b46712b6" (UID: "3472e0a8-d76d-45af-a9f4-cae9b46712b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:40.715800 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.715743 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq5bv\" (UniqueName: \"kubernetes.io/projected/3472e0a8-d76d-45af-a9f4-cae9b46712b6-kube-api-access-wq5bv\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:40.715800 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.715766 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:40.715800 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.715776 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:40.715800 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.715784 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3472e0a8-d76d-45af-a9f4-cae9b46712b6-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:40.855204 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.855181 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" Oct 30 18:22:40.855204 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.855187 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx" event={"ID":"904c45e3-164e-45ee-b877-b8b565fbb7fd","Type":"ContainerDied","Data":"5dffc454bc20a3d2e164ffc8e8f27b0e9e12de073278df1b306ff106f21be0e6"} Oct 30 18:22:40.855714 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.855240 4603 scope.go:122] "RemoveContainer" containerID="c24b5036c142521f0dd0323bb8c916177cfe4f0304d07a6eecbb0055150a4689" Oct 30 18:22:40.857133 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.857113 4603 generic.go:356] "Generic (PLEG): container finished" podID="3472e0a8-d76d-45af-a9f4-cae9b46712b6" containerID="0bccd0424b6f1d37697cbb71196d9656ae3e70d05810e8f2716bc744e3c2db48" exitCode=0 Oct 30 18:22:40.857236 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.857178 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" Oct 30 18:22:40.857236 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.857195 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" event={"ID":"3472e0a8-d76d-45af-a9f4-cae9b46712b6","Type":"ContainerDied","Data":"0bccd0424b6f1d37697cbb71196d9656ae3e70d05810e8f2716bc744e3c2db48"} Oct 30 18:22:40.857319 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.857246 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb" event={"ID":"3472e0a8-d76d-45af-a9f4-cae9b46712b6","Type":"ContainerDied","Data":"60d4b160b41d53a21809cddae5f7d767a64d26564e3ad6588bffc7a379ca1691"} Oct 30 18:22:40.867148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.867129 4603 scope.go:122] "RemoveContainer" containerID="0bccd0424b6f1d37697cbb71196d9656ae3e70d05810e8f2716bc744e3c2db48" Oct 30 18:22:40.876317 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.876298 4603 scope.go:122] "RemoveContainer" containerID="914b115695314e21fb1df670d9d1d9d69a1c192160a66d27a4fa26c6ae1dafbf" Oct 30 18:22:40.885247 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.885234 4603 scope.go:122] "RemoveContainer" containerID="0bccd0424b6f1d37697cbb71196d9656ae3e70d05810e8f2716bc744e3c2db48" Oct 30 18:22:40.885546 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:22:40.885491 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bccd0424b6f1d37697cbb71196d9656ae3e70d05810e8f2716bc744e3c2db48\": container with ID starting with 0bccd0424b6f1d37697cbb71196d9656ae3e70d05810e8f2716bc744e3c2db48 not found: ID does not exist" containerID="0bccd0424b6f1d37697cbb71196d9656ae3e70d05810e8f2716bc744e3c2db48" Oct 30 18:22:40.885595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.885555 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bccd0424b6f1d37697cbb71196d9656ae3e70d05810e8f2716bc744e3c2db48"} err="failed to get container status \"0bccd0424b6f1d37697cbb71196d9656ae3e70d05810e8f2716bc744e3c2db48\": rpc error: code = NotFound desc = could not find container \"0bccd0424b6f1d37697cbb71196d9656ae3e70d05810e8f2716bc744e3c2db48\": container with ID starting with 0bccd0424b6f1d37697cbb71196d9656ae3e70d05810e8f2716bc744e3c2db48 not found: ID does not exist" Oct 30 18:22:40.885595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.885571 4603 scope.go:122] "RemoveContainer" containerID="914b115695314e21fb1df670d9d1d9d69a1c192160a66d27a4fa26c6ae1dafbf" Oct 30 18:22:40.885802 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:22:40.885789 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"914b115695314e21fb1df670d9d1d9d69a1c192160a66d27a4fa26c6ae1dafbf\": container with ID starting with 914b115695314e21fb1df670d9d1d9d69a1c192160a66d27a4fa26c6ae1dafbf not found: ID does not exist" containerID="914b115695314e21fb1df670d9d1d9d69a1c192160a66d27a4fa26c6ae1dafbf" Oct 30 18:22:40.885843 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.885809 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"914b115695314e21fb1df670d9d1d9d69a1c192160a66d27a4fa26c6ae1dafbf"} err="failed to get container status \"914b115695314e21fb1df670d9d1d9d69a1c192160a66d27a4fa26c6ae1dafbf\": rpc error: code = NotFound desc = could not find container \"914b115695314e21fb1df670d9d1d9d69a1c192160a66d27a4fa26c6ae1dafbf\": container with ID starting with 914b115695314e21fb1df670d9d1d9d69a1c192160a66d27a4fa26c6ae1dafbf not found: ID does not exist" Oct 30 18:22:40.913113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.913089 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx"] Oct 30 18:22:40.916473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.916455 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a8dc70-openshift-default-55d896d7f9-jr2lx"] Oct 30 18:22:40.950227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.950205 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb"] Oct 30 18:22:40.953127 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:40.953111 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-069812-openshift-default-694cb7645c-phdtb"] Oct 30 18:22:41.251705 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:41.251669 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:41.411487 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:41.411456 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3472e0a8-d76d-45af-a9f4-cae9b46712b6" path="/var/lib/kubelet/pods/3472e0a8-d76d-45af-a9f4-cae9b46712b6/volumes" Oct 30 18:22:41.412112 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:41.412095 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="904c45e3-164e-45ee-b877-b8b565fbb7fd" path="/var/lib/kubelet/pods/904c45e3-164e-45ee-b877-b8b565fbb7fd/volumes" Oct 30 18:22:41.418529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:41.418490 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:41.570726 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:41.570690 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:41.725374 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:41.725289 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:42.936469 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:42.936433 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:43.096559 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:43.096363 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:43.340526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:43.336162 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:43.856171 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:43.856141 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:44.048530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:44.046263 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:44.509318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:44.509276 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:45.138247 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:45.138214 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:45.281074 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:45.281044 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:46.106225 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:46.106192 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:47.825412 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:47.825382 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85" Oct 30 18:22:48.239714 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:48.239613 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:49.815403 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:49.815367 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:49.988533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:49.987639 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:51.404117 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:51.404086 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:51.590351 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:51.590286 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:51.735903 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:51.735804 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:51.883523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:51.883100 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:52.099630 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:22:52.097982 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:22:53.329043 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:53.329006 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:53.410012 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:53.409984 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv"] Oct 30 18:22:53.410221 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:53.410198 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv" podUID="0f4b207b-5280-4361-9e36-a64feaf49cff" containerName="payload-processing" containerID="cri-o://082a296c35177610e0222286fd6d9c789ef0a19ea73b70bedb29469dbbe1e880" gracePeriod=30 Oct 30 18:22:53.426629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:53.426606 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt"] Oct 30 18:22:53.426809 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:53.426792 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt" podUID="74762a1d-083f-43ed-abb4-8ea80c4ba83c" containerName="payload-pre-processing" containerID="cri-o://99f31cc41833897c405c9af8d272a3c45ed05a2a5e376727a4d6efc6891a3c2d" gracePeriod=30 Oct 30 18:22:53.521654 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:53.521621 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:53.660567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:53.660441 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:53.859104 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:53.858722 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv" Oct 30 18:22:54.024528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:53.997164 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f4b207b-5280-4361-9e36-a64feaf49cff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4b207b-5280-4361-9e36-a64feaf49cff-plugins-config-volume\") pod \"0f4b207b-5280-4361-9e36-a64feaf49cff\" (UID: \"0f4b207b-5280-4361-9e36-a64feaf49cff\") " Oct 30 18:22:54.024528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:53.997371 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f4b207b-5280-4361-9e36-a64feaf49cff-kube-api-access-v4n5n\" (UniqueName: \"kubernetes.io/projected/0f4b207b-5280-4361-9e36-a64feaf49cff-kube-api-access-v4n5n\") pod \"0f4b207b-5280-4361-9e36-a64feaf49cff\" (UID: \"0f4b207b-5280-4361-9e36-a64feaf49cff\") " Oct 30 18:22:54.024528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.021811 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:54.031562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.025389 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f4b207b-5280-4361-9e36-a64feaf49cff-plugins-config-volume" pod "0f4b207b-5280-4361-9e36-a64feaf49cff" (UID: "0f4b207b-5280-4361-9e36-a64feaf49cff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:54.031917 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.031878 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4b207b-5280-4361-9e36-a64feaf49cff-kube-api-access-v4n5n" pod "0f4b207b-5280-4361-9e36-a64feaf49cff" (UID: "0f4b207b-5280-4361-9e36-a64feaf49cff"). InnerVolumeSpecName "kube-api-access-v4n5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:54.033386 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.033345 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt" Oct 30 18:22:54.041260 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.041227 4603 generic.go:356] "Generic (PLEG): container finished" podID="74762a1d-083f-43ed-abb4-8ea80c4ba83c" containerID="99f31cc41833897c405c9af8d272a3c45ed05a2a5e376727a4d6efc6891a3c2d" exitCode=0 Oct 30 18:22:54.041379 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.041338 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt" event={"ID":"74762a1d-083f-43ed-abb4-8ea80c4ba83c","Type":"ContainerDied","Data":"99f31cc41833897c405c9af8d272a3c45ed05a2a5e376727a4d6efc6891a3c2d"} Oct 30 18:22:54.041440 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.041409 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt" event={"ID":"74762a1d-083f-43ed-abb4-8ea80c4ba83c","Type":"ContainerDied","Data":"ac249cf57eaea495bc9eb10decb15bd0ba502c97bf16c45d538ad472297cc4f0"} Oct 30 18:22:54.041490 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.041445 4603 scope.go:122] "RemoveContainer" containerID="99f31cc41833897c405c9af8d272a3c45ed05a2a5e376727a4d6efc6891a3c2d" Oct 30 18:22:54.041759 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.041662 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt" Oct 30 18:22:54.042962 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.042943 4603 generic.go:356] "Generic (PLEG): container finished" podID="0f4b207b-5280-4361-9e36-a64feaf49cff" containerID="082a296c35177610e0222286fd6d9c789ef0a19ea73b70bedb29469dbbe1e880" exitCode=0 Oct 30 18:22:54.043095 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.043072 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv" Oct 30 18:22:54.043160 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.043089 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv" event={"ID":"0f4b207b-5280-4361-9e36-a64feaf49cff","Type":"ContainerDied","Data":"082a296c35177610e0222286fd6d9c789ef0a19ea73b70bedb29469dbbe1e880"} Oct 30 18:22:54.043160 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.043135 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv" event={"ID":"0f4b207b-5280-4361-9e36-a64feaf49cff","Type":"ContainerDied","Data":"64fddb07734f8041f16456ec15006aa304437f5a837b1d13d70891b5c1136168"} Oct 30 18:22:54.054390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.054375 4603 scope.go:122] "RemoveContainer" containerID="99f31cc41833897c405c9af8d272a3c45ed05a2a5e376727a4d6efc6891a3c2d" Oct 30 18:22:54.054693 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:22:54.054675 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99f31cc41833897c405c9af8d272a3c45ed05a2a5e376727a4d6efc6891a3c2d\": container with ID starting with 99f31cc41833897c405c9af8d272a3c45ed05a2a5e376727a4d6efc6891a3c2d not found: ID does not exist" containerID="99f31cc41833897c405c9af8d272a3c45ed05a2a5e376727a4d6efc6891a3c2d" Oct 30 18:22:54.054765 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.054700 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99f31cc41833897c405c9af8d272a3c45ed05a2a5e376727a4d6efc6891a3c2d"} err="failed to get container status \"99f31cc41833897c405c9af8d272a3c45ed05a2a5e376727a4d6efc6891a3c2d\": rpc error: code = NotFound desc = could not find container \"99f31cc41833897c405c9af8d272a3c45ed05a2a5e376727a4d6efc6891a3c2d\": container with ID starting with 99f31cc41833897c405c9af8d272a3c45ed05a2a5e376727a4d6efc6891a3c2d not found: ID does not exist" Oct 30 18:22:54.054765 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.054717 4603 scope.go:122] "RemoveContainer" containerID="082a296c35177610e0222286fd6d9c789ef0a19ea73b70bedb29469dbbe1e880" Oct 30 18:22:54.063470 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.063451 4603 scope.go:122] "RemoveContainer" containerID="082a296c35177610e0222286fd6d9c789ef0a19ea73b70bedb29469dbbe1e880" Oct 30 18:22:54.063898 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:22:54.063865 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"082a296c35177610e0222286fd6d9c789ef0a19ea73b70bedb29469dbbe1e880\": container with ID starting with 082a296c35177610e0222286fd6d9c789ef0a19ea73b70bedb29469dbbe1e880 not found: ID does not exist" containerID="082a296c35177610e0222286fd6d9c789ef0a19ea73b70bedb29469dbbe1e880" Oct 30 18:22:54.063980 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.063895 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"082a296c35177610e0222286fd6d9c789ef0a19ea73b70bedb29469dbbe1e880"} err="failed to get container status \"082a296c35177610e0222286fd6d9c789ef0a19ea73b70bedb29469dbbe1e880\": rpc error: code = NotFound desc = could not find container \"082a296c35177610e0222286fd6d9c789ef0a19ea73b70bedb29469dbbe1e880\": container with ID starting with 082a296c35177610e0222286fd6d9c789ef0a19ea73b70bedb29469dbbe1e880 not found: ID does not exist" Oct 30 18:22:54.099367 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.099347 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74762a1d-083f-43ed-abb4-8ea80c4ba83c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74762a1d-083f-43ed-abb4-8ea80c4ba83c-plugins-config-volume\") pod \"74762a1d-083f-43ed-abb4-8ea80c4ba83c\" (UID: \"74762a1d-083f-43ed-abb4-8ea80c4ba83c\") " Oct 30 18:22:54.099476 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.099418 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74762a1d-083f-43ed-abb4-8ea80c4ba83c-kube-api-access-s99lx\" (UniqueName: \"kubernetes.io/projected/74762a1d-083f-43ed-abb4-8ea80c4ba83c-kube-api-access-s99lx\") pod \"74762a1d-083f-43ed-abb4-8ea80c4ba83c\" (UID: \"74762a1d-083f-43ed-abb4-8ea80c4ba83c\") " Oct 30 18:22:54.099705 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.099683 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74762a1d-083f-43ed-abb4-8ea80c4ba83c-plugins-config-volume" pod "74762a1d-083f-43ed-abb4-8ea80c4ba83c" (UID: "74762a1d-083f-43ed-abb4-8ea80c4ba83c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:54.099999 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.099984 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4b207b-5280-4361-9e36-a64feaf49cff-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:54.100057 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.100003 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74762a1d-083f-43ed-abb4-8ea80c4ba83c-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:54.100057 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.100012 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4n5n\" (UniqueName: \"kubernetes.io/projected/0f4b207b-5280-4361-9e36-a64feaf49cff-kube-api-access-v4n5n\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:54.101629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.101607 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74762a1d-083f-43ed-abb4-8ea80c4ba83c-kube-api-access-s99lx" pod "74762a1d-083f-43ed-abb4-8ea80c4ba83c" (UID: "74762a1d-083f-43ed-abb4-8ea80c4ba83c"). InnerVolumeSpecName "kube-api-access-s99lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:54.127309 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.127263 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv"] Oct 30 18:22:54.133498 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.133477 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d951c4-b7d47848b-vjnlv"] Oct 30 18:22:54.170219 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.170193 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:54.231526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.228145 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s99lx\" (UniqueName: \"kubernetes.io/projected/74762a1d-083f-43ed-abb4-8ea80c4ba83c-kube-api-access-s99lx\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:22:54.322602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.322573 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:54.386842 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.386776 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt"] Oct 30 18:22:54.392698 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.392665 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d951c4-585867c47f-xwhmt"] Oct 30 18:22:54.652855 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:54.652770 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:55.410816 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:55.410783 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f4b207b-5280-4361-9e36-a64feaf49cff" path="/var/lib/kubelet/pods/0f4b207b-5280-4361-9e36-a64feaf49cff/volumes" Oct 30 18:22:55.411225 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:55.411209 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74762a1d-083f-43ed-abb4-8ea80c4ba83c" path="/var/lib/kubelet/pods/74762a1d-083f-43ed-abb4-8ea80c4ba83c/volumes" Oct 30 18:22:56.051006 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:56.050974 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:57.134368 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:57.134335 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:57.928536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:57.928471 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:58.062093 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:58.062056 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:59.478584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.478553 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p"] Oct 30 18:22:59.479106 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.479093 4603 cpu_manager.go:414] "containerMap: removing container" podUID="3472e0a8-d76d-45af-a9f4-cae9b46712b6" containerName="istio-proxy" Oct 30 18:22:59.479151 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.479108 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3472e0a8-d76d-45af-a9f4-cae9b46712b6" containerName="istio-proxy" Oct 30 18:22:59.479151 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.479132 4603 cpu_manager.go:414] "containerMap: removing container" podUID="0f4b207b-5280-4361-9e36-a64feaf49cff" containerName="payload-processing" Oct 30 18:22:59.479151 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.479137 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f4b207b-5280-4361-9e36-a64feaf49cff" containerName="payload-processing" Oct 30 18:22:59.479271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.479156 4603 cpu_manager.go:414] "containerMap: removing container" podUID="904c45e3-164e-45ee-b877-b8b565fbb7fd" containerName="istio-proxy" Oct 30 18:22:59.479271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.479160 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="904c45e3-164e-45ee-b877-b8b565fbb7fd" containerName="istio-proxy" Oct 30 18:22:59.479271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.479171 4603 cpu_manager.go:414] "containerMap: removing container" podUID="74762a1d-083f-43ed-abb4-8ea80c4ba83c" containerName="payload-pre-processing" Oct 30 18:22:59.479271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.479175 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74762a1d-083f-43ed-abb4-8ea80c4ba83c" containerName="payload-pre-processing" Oct 30 18:22:59.479271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.479187 4603 cpu_manager.go:414] "containerMap: removing container" podUID="904c45e3-164e-45ee-b877-b8b565fbb7fd" containerName="istio-proxy" Oct 30 18:22:59.479271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.479191 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="904c45e3-164e-45ee-b877-b8b565fbb7fd" containerName="istio-proxy" Oct 30 18:22:59.479271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.479272 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f4b207b-5280-4361-9e36-a64feaf49cff" containerName="payload-processing" Oct 30 18:22:59.479556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.479284 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="904c45e3-164e-45ee-b877-b8b565fbb7fd" containerName="istio-proxy" Oct 30 18:22:59.479556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.479291 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="3472e0a8-d76d-45af-a9f4-cae9b46712b6" containerName="istio-proxy" Oct 30 18:22:59.479556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.479297 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="74762a1d-083f-43ed-abb4-8ea80c4ba83c" containerName="payload-pre-processing" Oct 30 18:22:59.479556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.479303 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="904c45e3-164e-45ee-b877-b8b565fbb7fd" containerName="istio-proxy" Oct 30 18:22:59.483995 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.483980 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" Oct 30 18:22:59.498124 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.498102 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-ca5f33-serving-cert\"" Oct 30 18:22:59.598449 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.598419 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc2q9\" (UniqueName: \"kubernetes.io/projected/6109f83b-e952-42b8-bd57-c7f53f8adadb-kube-api-access-zc2q9\") pod \"maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p\" (UID: \"6109f83b-e952-42b8-bd57-c7f53f8adadb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" Oct 30 18:22:59.598660 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.598638 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6109f83b-e952-42b8-bd57-c7f53f8adadb-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p\" (UID: \"6109f83b-e952-42b8-bd57-c7f53f8adadb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" Oct 30 18:22:59.598750 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.598691 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6109f83b-e952-42b8-bd57-c7f53f8adadb-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p\" (UID: \"6109f83b-e952-42b8-bd57-c7f53f8adadb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" Oct 30 18:22:59.598975 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.598956 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6109f83b-e952-42b8-bd57-c7f53f8adadb-metrics-certs\") pod \"maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p\" (UID: \"6109f83b-e952-42b8-bd57-c7f53f8adadb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" Oct 30 18:22:59.682795 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.682765 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:59.717528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.712659 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6109f83b-e952-42b8-bd57-c7f53f8adadb-metrics-certs\") pod \"maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p\" (UID: \"6109f83b-e952-42b8-bd57-c7f53f8adadb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" Oct 30 18:22:59.717528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.712843 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc2q9\" (UniqueName: \"kubernetes.io/projected/6109f83b-e952-42b8-bd57-c7f53f8adadb-kube-api-access-zc2q9\") pod \"maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p\" (UID: \"6109f83b-e952-42b8-bd57-c7f53f8adadb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" Oct 30 18:22:59.717528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.713051 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6109f83b-e952-42b8-bd57-c7f53f8adadb-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p\" (UID: \"6109f83b-e952-42b8-bd57-c7f53f8adadb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" Oct 30 18:22:59.717528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.713114 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6109f83b-e952-42b8-bd57-c7f53f8adadb-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p\" (UID: \"6109f83b-e952-42b8-bd57-c7f53f8adadb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" Oct 30 18:22:59.723525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.718047 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6109f83b-e952-42b8-bd57-c7f53f8adadb-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p\" (UID: \"6109f83b-e952-42b8-bd57-c7f53f8adadb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" Oct 30 18:22:59.733575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.731157 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6109f83b-e952-42b8-bd57-c7f53f8adadb-metrics-certs\") pod \"maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p\" (UID: \"6109f83b-e952-42b8-bd57-c7f53f8adadb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" Oct 30 18:22:59.737531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.736212 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6109f83b-e952-42b8-bd57-c7f53f8adadb-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p\" (UID: \"6109f83b-e952-42b8-bd57-c7f53f8adadb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" Oct 30 18:22:59.798761 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.798631 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc2q9\" (UniqueName: \"kubernetes.io/projected/6109f83b-e952-42b8-bd57-c7f53f8adadb-kube-api-access-zc2q9\") pod \"maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p\" (UID: \"6109f83b-e952-42b8-bd57-c7f53f8adadb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" Oct 30 18:22:59.799062 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.798963 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" Oct 30 18:22:59.833320 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:22:59.833289 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:01.163803 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:01.163769 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:01.306149 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:01.306116 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:01.449039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:01.448960 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww"] Oct 30 18:23:01.449247 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:01.449209 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" podUID="30b23750-56cb-4895-80a8-8e54ae819f7a" containerName="istio-proxy" containerID="cri-o://801170481af59c9f323d8a8822629a2f1faae32a72e425359bd87149acfddb70" gracePeriod=30 Oct 30 18:23:01.499305 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:01.499277 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:01.729598 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:01.723637 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:02.917257 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:02.917181 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz"] Oct 30 18:23:03.119033 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:03.119005 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:03.272319 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:03.272287 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:03.559082 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:03.559005 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:05.328106 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:05.328071 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:06.725167 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.725107 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:23:06.844332 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.844302 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-podinfo\") pod \"30b23750-56cb-4895-80a8-8e54ae819f7a\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " Oct 30 18:23:06.844565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.844346 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-workload-certs\") pod \"30b23750-56cb-4895-80a8-8e54ae819f7a\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " Oct 30 18:23:06.844565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.844381 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-envoy\") pod \"30b23750-56cb-4895-80a8-8e54ae819f7a\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " Oct 30 18:23:06.844565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.844425 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30b23750-56cb-4895-80a8-8e54ae819f7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b23750-56cb-4895-80a8-8e54ae819f7a-istiod-ca-cert\") pod \"30b23750-56cb-4895-80a8-8e54ae819f7a\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " Oct 30 18:23:06.844565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.844452 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-token\" (UniqueName: \"kubernetes.io/projected/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-token\") pod \"30b23750-56cb-4895-80a8-8e54ae819f7a\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " Oct 30 18:23:06.844565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.844476 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-workload-socket\") pod \"30b23750-56cb-4895-80a8-8e54ae819f7a\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " Oct 30 18:23:06.844864 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.844569 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b23750-56cb-4895-80a8-8e54ae819f7a-kube-api-access-n6vwk\" (UniqueName: \"kubernetes.io/projected/30b23750-56cb-4895-80a8-8e54ae819f7a-kube-api-access-n6vwk\") pod \"30b23750-56cb-4895-80a8-8e54ae819f7a\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " Oct 30 18:23:06.844864 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.844614 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-credential-socket\") pod \"30b23750-56cb-4895-80a8-8e54ae819f7a\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " Oct 30 18:23:06.844864 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.844726 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-workload-certs" pod "30b23750-56cb-4895-80a8-8e54ae819f7a" (UID: "30b23750-56cb-4895-80a8-8e54ae819f7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:06.844864 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.844789 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-data\") pod \"30b23750-56cb-4895-80a8-8e54ae819f7a\" (UID: \"30b23750-56cb-4895-80a8-8e54ae819f7a\") " Oct 30 18:23:06.845249 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.844905 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b23750-56cb-4895-80a8-8e54ae819f7a-istiod-ca-cert" pod "30b23750-56cb-4895-80a8-8e54ae819f7a" (UID: "30b23750-56cb-4895-80a8-8e54ae819f7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:06.845249 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.845045 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-workload-socket" pod "30b23750-56cb-4895-80a8-8e54ae819f7a" (UID: "30b23750-56cb-4895-80a8-8e54ae819f7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:06.845368 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.845255 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-data" pod "30b23750-56cb-4895-80a8-8e54ae819f7a" (UID: "30b23750-56cb-4895-80a8-8e54ae819f7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:06.845451 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.845417 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-credential-socket" pod "30b23750-56cb-4895-80a8-8e54ae819f7a" (UID: "30b23750-56cb-4895-80a8-8e54ae819f7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:06.845731 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.845708 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:23:06.845802 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.845738 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:23:06.845802 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.845754 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b23750-56cb-4895-80a8-8e54ae819f7a-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:23:06.845802 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.845767 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:23:06.845802 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.845782 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:23:06.846897 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.846869 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-envoy" pod "30b23750-56cb-4895-80a8-8e54ae819f7a" (UID: "30b23750-56cb-4895-80a8-8e54ae819f7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:06.847008 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.846953 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-token" pod "30b23750-56cb-4895-80a8-8e54ae819f7a" (UID: "30b23750-56cb-4895-80a8-8e54ae819f7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:06.847008 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.846981 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-podinfo" pod "30b23750-56cb-4895-80a8-8e54ae819f7a" (UID: "30b23750-56cb-4895-80a8-8e54ae819f7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:23:06.847008 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.846981 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b23750-56cb-4895-80a8-8e54ae819f7a-kube-api-access-n6vwk" pod "30b23750-56cb-4895-80a8-8e54ae819f7a" (UID: "30b23750-56cb-4895-80a8-8e54ae819f7a"). InnerVolumeSpecName "kube-api-access-n6vwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:06.946860 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.946834 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:23:06.946860 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.946860 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6vwk\" (UniqueName: \"kubernetes.io/projected/30b23750-56cb-4895-80a8-8e54ae819f7a-kube-api-access-n6vwk\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:23:06.947013 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.946873 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:23:06.947013 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:06.946885 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b23750-56cb-4895-80a8-8e54ae819f7a-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:23:07.078448 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:07.078419 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:07.163692 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:07.163572 4603 generic.go:356] "Generic (PLEG): container finished" podID="30b23750-56cb-4895-80a8-8e54ae819f7a" containerID="801170481af59c9f323d8a8822629a2f1faae32a72e425359bd87149acfddb70" exitCode=0 Oct 30 18:23:07.164525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:07.164074 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" event={"ID":"30b23750-56cb-4895-80a8-8e54ae819f7a","Type":"ContainerDied","Data":"801170481af59c9f323d8a8822629a2f1faae32a72e425359bd87149acfddb70"} Oct 30 18:23:07.164525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:07.164137 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" event={"ID":"30b23750-56cb-4895-80a8-8e54ae819f7a","Type":"ContainerDied","Data":"d961ccfad98a23d4dd7ccaeaf941e44879fac714c046500a707d6dac0e49105c"} Oct 30 18:23:07.164525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:07.164168 4603 scope.go:122] "RemoveContainer" containerID="801170481af59c9f323d8a8822629a2f1faae32a72e425359bd87149acfddb70" Oct 30 18:23:07.164525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:07.164343 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww" Oct 30 18:23:07.200767 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:07.200741 4603 scope.go:122] "RemoveContainer" containerID="00c0b21e487800fbcf8c54e46f87001a571fa1f9f79e2f0dcbd0d555ec3b60cb" Oct 30 18:23:07.229887 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:07.229863 4603 scope.go:122] "RemoveContainer" containerID="801170481af59c9f323d8a8822629a2f1faae32a72e425359bd87149acfddb70" Oct 30 18:23:07.231225 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:23:07.230342 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"801170481af59c9f323d8a8822629a2f1faae32a72e425359bd87149acfddb70\": container with ID starting with 801170481af59c9f323d8a8822629a2f1faae32a72e425359bd87149acfddb70 not found: ID does not exist" containerID="801170481af59c9f323d8a8822629a2f1faae32a72e425359bd87149acfddb70" Oct 30 18:23:07.231225 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:07.230386 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"801170481af59c9f323d8a8822629a2f1faae32a72e425359bd87149acfddb70"} err="failed to get container status \"801170481af59c9f323d8a8822629a2f1faae32a72e425359bd87149acfddb70\": rpc error: code = NotFound desc = could not find container \"801170481af59c9f323d8a8822629a2f1faae32a72e425359bd87149acfddb70\": container with ID starting with 801170481af59c9f323d8a8822629a2f1faae32a72e425359bd87149acfddb70 not found: ID does not exist" Oct 30 18:23:07.231225 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:07.230414 4603 scope.go:122] "RemoveContainer" containerID="00c0b21e487800fbcf8c54e46f87001a571fa1f9f79e2f0dcbd0d555ec3b60cb" Oct 30 18:23:07.231225 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:23:07.231148 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00c0b21e487800fbcf8c54e46f87001a571fa1f9f79e2f0dcbd0d555ec3b60cb\": container with ID starting with 00c0b21e487800fbcf8c54e46f87001a571fa1f9f79e2f0dcbd0d555ec3b60cb not found: ID does not exist" containerID="00c0b21e487800fbcf8c54e46f87001a571fa1f9f79e2f0dcbd0d555ec3b60cb" Oct 30 18:23:07.231225 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:07.231178 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c0b21e487800fbcf8c54e46f87001a571fa1f9f79e2f0dcbd0d555ec3b60cb"} err="failed to get container status \"00c0b21e487800fbcf8c54e46f87001a571fa1f9f79e2f0dcbd0d555ec3b60cb\": rpc error: code = NotFound desc = could not find container \"00c0b21e487800fbcf8c54e46f87001a571fa1f9f79e2f0dcbd0d555ec3b60cb\": container with ID starting with 00c0b21e487800fbcf8c54e46f87001a571fa1f9f79e2f0dcbd0d555ec3b60cb not found: ID does not exist" Oct 30 18:23:07.235521 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:07.235487 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww"] Oct 30 18:23:07.238570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:07.238552 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d951c4-openshift-default-9f8559cf-5w9ww"] Oct 30 18:23:07.410351 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:07.410291 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30b23750-56cb-4895-80a8-8e54ae819f7a" path="/var/lib/kubelet/pods/30b23750-56cb-4895-80a8-8e54ae819f7a/volumes" Oct 30 18:23:07.917794 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:07.917759 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz" podUID="58abeaf7-cdae-438a-a1e3-276aa3e76393" containerName="payload-processing" containerID="cri-o://427038c55cc4abe29cb9b28cdcd5ce3cedc61434760160b3b1384de68cdad8dc" gracePeriod=25 Oct 30 18:23:08.174405 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:08.174347 4603 generic.go:356] "Generic (PLEG): container finished" podID="58abeaf7-cdae-438a-a1e3-276aa3e76393" containerID="427038c55cc4abe29cb9b28cdcd5ce3cedc61434760160b3b1384de68cdad8dc" exitCode=0 Oct 30 18:23:08.174562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:08.174423 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz" event={"ID":"58abeaf7-cdae-438a-a1e3-276aa3e76393","Type":"ContainerDied","Data":"427038c55cc4abe29cb9b28cdcd5ce3cedc61434760160b3b1384de68cdad8dc"} Oct 30 18:23:08.174562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:08.174482 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz" event={"ID":"58abeaf7-cdae-438a-a1e3-276aa3e76393","Type":"ContainerDied","Data":"2c716334a4b338893c42ce8924a30e07906c313ec70b67916e87ce775179e849"} Oct 30 18:23:08.174562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:08.174523 4603 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c716334a4b338893c42ce8924a30e07906c313ec70b67916e87ce775179e849" Oct 30 18:23:08.184729 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:08.184700 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz" Oct 30 18:23:08.374792 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:08.374748 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58abeaf7-cdae-438a-a1e3-276aa3e76393-kube-api-access-kh9jk\" (UniqueName: \"kubernetes.io/projected/58abeaf7-cdae-438a-a1e3-276aa3e76393-kube-api-access-kh9jk\") pod \"58abeaf7-cdae-438a-a1e3-276aa3e76393\" (UID: \"58abeaf7-cdae-438a-a1e3-276aa3e76393\") " Oct 30 18:23:08.374946 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:08.374867 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58abeaf7-cdae-438a-a1e3-276aa3e76393-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58abeaf7-cdae-438a-a1e3-276aa3e76393-plugins-config-volume\") pod \"58abeaf7-cdae-438a-a1e3-276aa3e76393\" (UID: \"58abeaf7-cdae-438a-a1e3-276aa3e76393\") " Oct 30 18:23:08.375240 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:08.375210 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58abeaf7-cdae-438a-a1e3-276aa3e76393-plugins-config-volume" pod "58abeaf7-cdae-438a-a1e3-276aa3e76393" (UID: "58abeaf7-cdae-438a-a1e3-276aa3e76393"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:08.375426 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:08.375412 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58abeaf7-cdae-438a-a1e3-276aa3e76393-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:23:08.377006 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:08.376986 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58abeaf7-cdae-438a-a1e3-276aa3e76393-kube-api-access-kh9jk" pod "58abeaf7-cdae-438a-a1e3-276aa3e76393" (UID: "58abeaf7-cdae-438a-a1e3-276aa3e76393"). InnerVolumeSpecName "kube-api-access-kh9jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:08.476552 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:08.476465 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh9jk\" (UniqueName: \"kubernetes.io/projected/58abeaf7-cdae-438a-a1e3-276aa3e76393-kube-api-access-kh9jk\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:23:09.180181 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:09.180151 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz" Oct 30 18:23:09.230908 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:09.230882 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz"] Oct 30 18:23:09.242531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:09.242486 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e0bc49-76544fffcd-fxqwz"] Oct 30 18:23:09.259004 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:09.258978 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:09.468758 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:09.466820 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58abeaf7-cdae-438a-a1e3-276aa3e76393" path="/var/lib/kubelet/pods/58abeaf7-cdae-438a-a1e3-276aa3e76393/volumes" Oct 30 18:23:10.839130 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:10.839093 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:12.552148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:12.552109 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:14.457068 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:14.457037 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:15.957945 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:15.957910 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:16.242653 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:16.242530 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:17.831963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:17.831924 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:17.995567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:17.986113 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:19.121904 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.121874 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:19.517999 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.517935 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t"] Oct 30 18:23:19.518484 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.518471 4603 cpu_manager.go:414] "containerMap: removing container" podUID="58abeaf7-cdae-438a-a1e3-276aa3e76393" containerName="payload-processing" Oct 30 18:23:19.518558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.518486 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58abeaf7-cdae-438a-a1e3-276aa3e76393" containerName="payload-processing" Oct 30 18:23:19.518558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.518529 4603 cpu_manager.go:414] "containerMap: removing container" podUID="3472e0a8-d76d-45af-a9f4-cae9b46712b6" containerName="istio-proxy" Oct 30 18:23:19.518558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.518536 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3472e0a8-d76d-45af-a9f4-cae9b46712b6" containerName="istio-proxy" Oct 30 18:23:19.518558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.518546 4603 cpu_manager.go:414] "containerMap: removing container" podUID="30b23750-56cb-4895-80a8-8e54ae819f7a" containerName="istio-proxy" Oct 30 18:23:19.518558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.518552 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b23750-56cb-4895-80a8-8e54ae819f7a" containerName="istio-proxy" Oct 30 18:23:19.518716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.518564 4603 cpu_manager.go:414] "containerMap: removing container" podUID="30b23750-56cb-4895-80a8-8e54ae819f7a" containerName="istio-proxy" Oct 30 18:23:19.518716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.518569 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b23750-56cb-4895-80a8-8e54ae819f7a" containerName="istio-proxy" Oct 30 18:23:19.518716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.518662 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="3472e0a8-d76d-45af-a9f4-cae9b46712b6" containerName="istio-proxy" Oct 30 18:23:19.518716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.518675 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="58abeaf7-cdae-438a-a1e3-276aa3e76393" containerName="payload-processing" Oct 30 18:23:19.518716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.518684 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b23750-56cb-4895-80a8-8e54ae819f7a" containerName="istio-proxy" Oct 30 18:23:19.518716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.518691 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b23750-56cb-4895-80a8-8e54ae819f7a" containerName="istio-proxy" Oct 30 18:23:19.522060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.522033 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.541038 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.541017 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-7ffac8-openshift-default-dockercfg-8kwr2\"" Oct 30 18:23:19.543702 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.543681 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-workload-socket\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.543801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.543736 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-workload-certs\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.543848 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.543809 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa0c0084-b530-436d-936a-b94ebc16fe03-istiod-ca-cert\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.543942 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.543923 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-envoy\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.544012 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.543996 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwqhx\" (UniqueName: \"kubernetes.io/projected/fa0c0084-b530-436d-936a-b94ebc16fe03-kube-api-access-rwqhx\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.544086 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.544072 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-podinfo\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.544127 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.544095 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-data\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.544168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.544126 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-token\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.544168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.544161 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-credential-socket\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.601195 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.601167 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:19.645206 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.645176 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-envoy\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.645353 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.645223 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwqhx\" (UniqueName: \"kubernetes.io/projected/fa0c0084-b530-436d-936a-b94ebc16fe03-kube-api-access-rwqhx\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.645353 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.645261 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-podinfo\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.645353 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.645279 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-data\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.645353 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.645312 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-token\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.645606 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.645364 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-credential-socket\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.645606 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.645434 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-workload-socket\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.645606 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.645522 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-workload-certs\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.645606 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.645562 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa0c0084-b530-436d-936a-b94ebc16fe03-istiod-ca-cert\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.645812 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.645749 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-data\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.645870 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.645833 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-credential-socket\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.646081 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.645870 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-workload-certs\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.646081 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.645926 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-workload-socket\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.646571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.646542 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa0c0084-b530-436d-936a-b94ebc16fe03-istiod-ca-cert\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.647740 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.647720 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-envoy\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.648019 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.648000 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-podinfo\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.668099 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.668075 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-token\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.668183 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.668148 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwqhx\" (UniqueName: \"kubernetes.io/projected/fa0c0084-b530-436d-936a-b94ebc16fe03-kube-api-access-rwqhx\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:19.765547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.764200 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:19.854917 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.854884 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-7ffac8-openshift-default-dockercfg-8kwr2\"" Oct 30 18:23:19.860616 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:19.860584 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:21.281023 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:21.280986 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:21.930184 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:23:21.930153 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:23:22.081286 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:22.081248 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:22.970724 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:22.970690 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:23.121819 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:23.121780 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:23.368949 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:23.368906 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:23.765685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:23.765612 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl"] Oct 30 18:23:23.771040 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:23.771021 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:23.787966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:23.787928 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-b8b654-openshift-default-dockercfg-lrm47\"" Oct 30 18:23:23.858578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:23.858550 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:23.895554 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:23.887026 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-workload-certs\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:23.895890 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:23.895855 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tphzm\" (UniqueName: \"kubernetes.io/projected/3e865ee8-6656-40a4-beca-34d6e43d4b8c-kube-api-access-tphzm\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:23.896048 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:23.896029 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-podinfo\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:23.896180 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:23.896166 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istiod-ca-cert\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:23.896382 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:23.896367 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-credential-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:23.896520 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:23.896486 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-workload-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:23.896695 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:23.896679 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-envoy\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:23.896836 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:23.896821 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-data\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:23.903544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:23.897078 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-token\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:24.006572 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.006530 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-token\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:24.007814 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.007765 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-workload-certs\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:24.008444 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.008426 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tphzm\" (UniqueName: \"kubernetes.io/projected/3e865ee8-6656-40a4-beca-34d6e43d4b8c-kube-api-access-tphzm\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:24.008837 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.008820 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-podinfo\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:24.009358 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.009341 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istiod-ca-cert\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:24.013522 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.010395 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-credential-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:24.013953 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.013938 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-workload-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:24.014394 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.014379 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-envoy\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:24.014879 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.014863 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-data\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:24.027620 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.010266 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istiod-ca-cert\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:24.028397 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.014291 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-workload-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:24.028582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.008359 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-workload-certs\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:24.028694 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.013876 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-credential-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:24.028794 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.015195 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-data\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:24.031200 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.031172 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:24.056543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.056489 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-envoy\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:24.058932 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.058888 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tphzm\" (UniqueName: \"kubernetes.io/projected/3e865ee8-6656-40a4-beca-34d6e43d4b8c-kube-api-access-tphzm\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:24.059660 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.059548 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-token\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:24.071079 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.071032 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-podinfo\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:24.088049 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.087991 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:24.384756 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.384673 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:24.588087 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.588056 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:24.740727 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.740648 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:24.941992 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:24.941730 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:25.134529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:25.132453 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:25.328873 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:25.328827 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:25.641904 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:25.641872 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:26.946422 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:26.946390 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:28.649041 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:28.648997 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:29.639777 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:29.639729 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t"] Oct 30 18:23:29.652945 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:29.652913 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl"] Oct 30 18:23:29.668912 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:29.668876 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p"] Oct 30 18:23:30.094011 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:30.093974 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t"] Oct 30 18:23:30.098836 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:30.098796 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:23:30.099005 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:30.098880 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:23:30.099005 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:30.098922 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:23:30.099548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:30.099450 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:23:30.099842 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:30.099541 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:23:30.099842 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:30.099589 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:23:30.105195 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:30.105169 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl"] Oct 30 18:23:30.112863 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:30.112786 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p"] Oct 30 18:23:30.114245 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:30.114220 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:30.118743 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:23:30.118715 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6109f83b_e952_42b8_bd57_c7f53f8adadb.slice/crio-38e4fc672cbd62cfc2b33823389aff4b23211e69ac01ca5cad988715d11a0fe9 WatchSource:0}: Error finding container 38e4fc672cbd62cfc2b33823389aff4b23211e69ac01ca5cad988715d11a0fe9: Status 404 returned error can't find the container with id 38e4fc672cbd62cfc2b33823389aff4b23211e69ac01ca5cad988715d11a0fe9 Oct 30 18:23:30.120293 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:30.120272 4603 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:23:30.455569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:30.453106 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" event={"ID":"3e865ee8-6656-40a4-beca-34d6e43d4b8c","Type":"ContainerStarted","Data":"927bde0819ecda487e1d05e1c7ec9ecb1f29a963e8991c98d10f681171bb61e1"} Oct 30 18:23:30.455569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:30.453185 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" event={"ID":"3e865ee8-6656-40a4-beca-34d6e43d4b8c","Type":"ContainerStarted","Data":"61464429c5d8e4a9143b2e3fdd59a48232b9db66b5fc90c31b6de6e9e4e557fa"} Oct 30 18:23:30.466215 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:30.466174 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" event={"ID":"fa0c0084-b530-436d-936a-b94ebc16fe03","Type":"ContainerStarted","Data":"bf290b19a878dbf3b3491b1985b16ba5030b0285dfeaf04cdee0238993694388"} Oct 30 18:23:30.466544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:30.466488 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" event={"ID":"fa0c0084-b530-436d-936a-b94ebc16fe03","Type":"ContainerStarted","Data":"528d53422376f33033805b667637dfa60ba7560eb55442cd355903fb6b30a8ac"} Oct 30 18:23:30.475013 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:30.474937 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" event={"ID":"6109f83b-e952-42b8-bd57-c7f53f8adadb","Type":"ContainerStarted","Data":"38e4fc672cbd62cfc2b33823389aff4b23211e69ac01ca5cad988715d11a0fe9"} Oct 30 18:23:30.510666 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:30.507782 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" podStartSLOduration=7.507764398 podStartE2EDuration="7.507764398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:30.506316918 +0000 UTC m=+1871.221886793" watchObservedRunningTime="2026-10-30 18:23:30.507764398 +0000 UTC m=+1871.223334274" Oct 30 18:23:30.689999 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:30.689958 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:30.864487 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:30.864077 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:30.865431 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:30.865295 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:31.090231 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:31.090146 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:31.095354 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:31.095318 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 18:23:31.095564 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:31.095392 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" podUID="3e865ee8-6656-40a4-beca-34d6e43d4b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 18:23:31.313828 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:31.313788 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:31.483064 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:31.482986 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" event={"ID":"6109f83b-e952-42b8-bd57-c7f53f8adadb","Type":"ContainerStarted","Data":"791c4bf544f35548f9338a77813978a9db37957c2f2855b024aaf39fffd7b650"} Oct 30 18:23:31.483064 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:31.483058 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" Oct 30 18:23:31.524857 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:31.524801 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" podStartSLOduration=12.524782053 podStartE2EDuration="12.524782053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:30.577853904 +0000 UTC m=+1871.293423837" watchObservedRunningTime="2026-10-30 18:23:31.524782053 +0000 UTC m=+1872.240351929" Oct 30 18:23:31.526961 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:31.526929 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" podStartSLOduration=32.134179148 podStartE2EDuration="32.526917621s" totalImagesPullingTime="392.738473ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:31.523489555 +0000 UTC m=+1872.239059429" watchObservedRunningTime="2026-10-30 18:23:31.526917621 +0000 UTC m=+1872.242487494" Oct 30 18:23:31.867979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:31.867950 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:31.975242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:31.975211 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:32.160016 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:32.159602 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:32.254842 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:32.191395 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:32.489027 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:32.488946 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:32.489202 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:32.489051 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:32.489523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:32.489489 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:23:32.489609 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:32.489565 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:23:32.799304 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:32.799264 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:33.336701 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:33.336670 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:33.489430 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:33.489396 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:34.536364 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:34.536333 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:34.846649 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:34.846572 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:35.163155 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:35.163052 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:35.315072 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:35.315040 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:36.183385 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:36.183354 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:36.457005 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:36.456934 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:23:37.494393 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:37.494368 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" Oct 30 18:23:37.966528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:37.962966 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:38.259398 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:38.259308 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:38.408156 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:38.408045 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:38.632536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:38.632485 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:39.631640 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:39.631607 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:39.805095 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:39.805064 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:40.162232 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.162182 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:40.236528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.222623 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:40.249528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.237262 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8"] Oct 30 18:23:40.287529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.276198 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:40.357744 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.357680 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:40.423582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.414015 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.529542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.513615 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:23:40.530004 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.515557 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjqcb\" (UniqueName: \"kubernetes.io/projected/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-kube-api-access-fjqcb\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.530811 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.530782 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.555425 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.555386 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.557808 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.557782 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.558130 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.558104 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.558313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.558289 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.558474 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.558455 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.558684 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.558665 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.558979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.558959 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.605257 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.605228 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-77db82-openshift-default-dockercfg-btt49\"" Oct 30 18:23:40.641280 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.624973 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:40.642254 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.642221 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:40.730584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.710053 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.803872 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.803840 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.804574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.804475 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.822621 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.822579 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8"] Oct 30 18:23:40.823163 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.805813 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.823163 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.806156 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.823362 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.757076 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:40.824400 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.823600 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.824400 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.823678 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.824400 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.823948 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqcb\" (UniqueName: \"kubernetes.io/projected/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-kube-api-access-fjqcb\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.824400 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.824066 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.824400 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.824121 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.824400 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.824170 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.824803 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.824728 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.825314 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.825099 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.825870 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.825847 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:40.853614 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:40.853580 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:41.138526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:41.125537 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:41.139070 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:41.139042 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:41.139317 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:41.139288 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:41.179296 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:41.179262 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:41.180060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:41.180033 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:41.180837 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:41.180817 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjqcb\" (UniqueName: \"kubernetes.io/projected/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-kube-api-access-fjqcb\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:41.239711 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:41.239670 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:23:41.242422 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:41.239945 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:23:41.282301 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:41.260200 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:23:41.282301 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:41.271722 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:23:41.473578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:41.470048 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:41.580530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:41.563394 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:23:41.674290 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:41.674264 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:23:41.719529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:41.705620 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:41.834611 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:41.834568 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:41.883529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:41.851238 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:23:42.262132 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:42.262091 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:42.366427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:42.269004 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:42.367199 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:42.295581 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:42.368046 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:42.295629 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:23:42.451025 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:42.377877 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:23:42.573873 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:42.570697 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:42.760558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:42.757591 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:23:42.760558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:42.758013 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:23:42.812644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:42.812612 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:23:42.835685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:42.835488 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:42.910825 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:42.876078 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:43.150884 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:43.043035 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:43.281645 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:43.281598 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:23:43.282903 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:43.282878 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:23:43.404583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:43.371332 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:43.437535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:43.406689 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:43.500747 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:43.429924 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:43.501456 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:43.432196 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:23:43.501697 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:43.501673 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" podUID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:43.576275 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:43.576232 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:43.761277 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:43.761175 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:43.799826 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:43.761521 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:23:43.825344 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:43.825312 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:23:43.881334 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:43.881287 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:23:44.101536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:44.089814 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:23:44.101536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:44.089884 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" podUID="22e0b280-2b65-4fad-8344-67049190d108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:44.156529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:44.156183 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:44.305529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:44.302106 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:23:44.405678 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:44.405581 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:23:44.476317 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:44.476269 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:44.568538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:44.567210 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:23:44.634076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:44.634025 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:44.650637 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:44.650587 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:23:44.811530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:44.807840 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:23:44.860548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:44.855625 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:44.920426 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:44.882136 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:45.125541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:45.123647 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8"] Oct 30 18:23:45.164219 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:45.164189 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:45.301982 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:45.301902 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:23:45.302210 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:45.302111 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:23:45.302284 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:45.302208 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:23:45.311544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:45.311104 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:45.409460 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:45.409367 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:23:45.499539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:45.492033 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:45.522573 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:45.519373 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:23:45.685614 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:45.681092 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:23:45.771714 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:45.747354 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Oct 30 18:23:45.771962 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:45.749279 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:45.850195 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:45.850159 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:45.879859 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:45.879822 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:45.888071 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:45.888039 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:23:45.919978 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:45.905899 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:23:46.332686 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.332594 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:46.444777 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.444020 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:46.581609 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.530804 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:46.656689 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:23:46.609336 4603 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.152s" Oct 30 18:23:46.656689 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.624371 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx"] Oct 30 18:23:46.673247 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.673198 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:46.752526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.751805 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx"] Oct 30 18:23:46.752526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.751861 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" event={"ID":"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af","Type":"ContainerStarted","Data":"2654115544eafc6601b4ef1d5975a68fb2baa2c63fdacb10f1848a04a2cf2028"} Oct 30 18:23:46.753200 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.753173 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:46.771633 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.771587 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:46.772253 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.772219 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:46.806373 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.787838 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-1fae38-openshift-default-dockercfg-4nkbk\"" Oct 30 18:23:46.827636 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.795314 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istiod-ca-cert\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:46.828118 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.828084 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhfdq\" (UniqueName: \"kubernetes.io/projected/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-kube-api-access-dhfdq\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:46.828413 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.828388 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-credential-socket\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:46.828598 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.828577 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-podinfo\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:46.828785 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.828768 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-workload-certs\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:46.828925 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.828908 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-envoy\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:46.829106 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.829088 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-workload-socket\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:46.829287 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.829270 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-token\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:46.829412 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.829396 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-data\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:46.907250 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.907165 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:46.926771 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.926709 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:23:46.927071 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.927038 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:23:46.964554 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.947671 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhfdq\" (UniqueName: \"kubernetes.io/projected/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-kube-api-access-dhfdq\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:46.968987 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.968952 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-credential-socket\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:46.981632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.981599 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-podinfo\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:46.981895 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.981874 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-workload-certs\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:46.982026 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.982012 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-envoy\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:46.982220 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.982203 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-workload-socket\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:46.998552 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.982393 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-token\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:46.998822 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.998795 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-data\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:46.999086 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.999068 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istiod-ca-cert\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:47.031542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.991987 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-workload-socket\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:47.031791 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.992935 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" event={"ID":"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af","Type":"ContainerStarted","Data":"c78b43d96281f3f140142fdc01deb094700395b4c1b15ddc1bc01e2b05f70932"} Oct 30 18:23:47.031967 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.991040 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-workload-certs\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:47.032062 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:47.007080 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-data\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:47.086608 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:46.971945 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-credential-socket\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:47.111615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:47.040700 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-envoy\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:47.111846 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:47.055352 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-podinfo\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:47.111945 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:47.056950 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istiod-ca-cert\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:47.155530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:47.120492 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhfdq\" (UniqueName: \"kubernetes.io/projected/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-kube-api-access-dhfdq\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:47.155821 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:47.139364 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:47.160577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:47.160227 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-token\") pod \"e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:47.247838 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:47.247797 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:47.363103 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:47.363058 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:47.706656 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:47.688623 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:23:47.706656 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:47.690093 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:23:47.706656 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:47.703702 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:23:47.711621 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:47.711591 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:23:47.711845 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:47.711824 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 18:23:47.793561 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:47.775060 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:23:47.814612 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:47.814580 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:47.833673 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:47.822614 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:23:47.909730 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:47.909670 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:47.935051 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:47.935007 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:23:47.935799 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:47.935767 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:48.042378 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:48.042348 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:48.060748 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:48.048992 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:48.145532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:48.094761 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:48.476578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:48.476030 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:23:48.476578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:48.476094 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 18:23:48.655610 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:48.631897 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:48.692170 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:48.692122 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:48.750626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:48.693269 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:23:48.830574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:48.786849 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:48.830574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:48.818313 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:48.830574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:48.829846 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:48.830574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:48.830108 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:23:48.836621 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:48.836586 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:48.914288 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:48.913760 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:23:48.941237 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:48.941180 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:23:48.941603 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:48.941531 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" podUID="b664f4ca-736e-4cb1-991d-985d1e6aff34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:48.941902 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:48.941802 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:48.943295 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:48.943225 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:49.073681 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:49.071624 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:23:49.073681 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:49.071688 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" podUID="4d6cd466-9f15-476d-9f04-6d03176f025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:49.509168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:49.504999 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:23:49.509168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:49.505084 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" podUID="d4a3adf6-fb23-4334-8bab-f800b4e8e089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:49.716608 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:49.711023 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:23:49.756529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:49.735954 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:49.816424 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:49.793948 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:49.835538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:49.825868 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:49.835538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:49.825918 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:49.861541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:49.853953 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:23:49.861541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:49.856179 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:23:49.925658 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:49.914696 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:23:50.046402 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:49.996670 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:50.046402 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:49.998208 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:23:50.046402 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:50.023474 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" podStartSLOduration=10.023456014 podStartE2EDuration="10.023456014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:47.140684388 +0000 UTC m=+1887.856254262" watchObservedRunningTime="2026-10-30 18:23:50.023456014 +0000 UTC m=+1890.739025888" Oct 30 18:23:50.094733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:50.081491 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:23:50.404629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:50.379611 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:50.597425 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:50.597376 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:23:50.611537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:50.603606 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:23:50.611841 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:50.611804 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:23:50.716634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:50.716574 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:23:50.749528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:50.747201 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:50.790255 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:50.790215 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx"] Oct 30 18:23:50.794618 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:50.794578 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:50.857864 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:50.857789 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:50.858642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:50.858613 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:51.093557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:51.039613 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:23:51.093557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:51.080593 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:23:51.093557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:51.089266 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:51.292553 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:51.268043 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" event={"ID":"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd","Type":"ContainerStarted","Data":"1e5af9b0ff55f75c72f909e4af6b746403e78714d59449f5456cb7814fc7de5f"} Oct 30 18:23:51.405821 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:51.351535 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:51.737746 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:51.737682 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:51.749680 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:51.749639 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:23:51.795060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:51.795010 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:51.867748 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:51.867701 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:51.868181 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:51.868140 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:23:52.049530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:52.040479 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:23:52.081529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:52.081256 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:52.107939 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:52.107885 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:52.552542 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:23:52.535218 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:23:52.552542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:52.538683 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" event={"ID":"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd","Type":"ContainerStarted","Data":"bb57dd61d380ff4d5f80723b7a4b6801e9620be7bdf496c960b11333446b6337"} Oct 30 18:23:52.648529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:52.621969 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" podStartSLOduration=7.621948486 podStartE2EDuration="7.621948486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:52.614862831 +0000 UTC m=+1893.330432708" watchObservedRunningTime="2026-10-30 18:23:52.621948486 +0000 UTC m=+1893.337518364" Oct 30 18:23:52.752989 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:52.752945 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:52.792640 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:52.791644 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:52.804903 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:52.803662 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:23:52.836398 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:52.836353 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:52.883875 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:52.883835 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:23:52.940693 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:52.940645 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:52.990962 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:52.990931 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:23:53.047530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:53.046513 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:53.100713 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:53.098751 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:23:53.144530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:53.142655 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:53.236712 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:53.236679 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:23:53.261119 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:53.245639 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:23:53.261119 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:53.260188 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:53.324362 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:53.324326 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:23:53.337570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:53.329893 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:23:53.346889 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:53.346864 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:53.465064 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:53.464996 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:23:53.469727 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:53.469702 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:23:53.562496 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:53.562461 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:53.564902 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:53.564878 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:23:54.472752 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:54.472712 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:55.241524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:55.241457 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:55.385208 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:55.385174 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:56.377239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:56.377205 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:56.616150 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:56.616118 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:57.535561 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:57.535524 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:57.687652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:57.687617 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:58.602535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:23:58.602426 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:00.341044 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:00.341010 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:01.685943 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:01.685909 4603 generic.go:356] "Generic (PLEG): container finished" podID="e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" containerID="f63162a2d6737b4440315e5f1da156b440d6a38db7e14306151fbc7d44df959a" exitCode=0 Oct 30 18:24:01.686413 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:01.685982 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" event={"ID":"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7","Type":"ContainerDied","Data":"f63162a2d6737b4440315e5f1da156b440d6a38db7e14306151fbc7d44df959a"} Oct 30 18:24:01.686413 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:01.686360 4603 scope.go:122] "RemoveContainer" containerID="f63162a2d6737b4440315e5f1da156b440d6a38db7e14306151fbc7d44df959a" Oct 30 18:24:01.687835 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:01.687809 4603 generic.go:356] "Generic (PLEG): container finished" podID="9d3f60a0-c196-48e7-9765-b1f3d8965367" containerID="69dadf26749f14c3df88d4f5b7b25f26514065a46b9b7b9e74baef4efec33e3c" exitCode=0 Oct 30 18:24:01.687949 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:01.687887 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" event={"ID":"9d3f60a0-c196-48e7-9765-b1f3d8965367","Type":"ContainerDied","Data":"69dadf26749f14c3df88d4f5b7b25f26514065a46b9b7b9e74baef4efec33e3c"} Oct 30 18:24:01.687997 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:01.687935 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:24:01.688042 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:01.688011 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:24:01.688091 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:01.688044 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:24:01.688236 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:01.688217 4603 scope.go:122] "RemoveContainer" containerID="69dadf26749f14c3df88d4f5b7b25f26514065a46b9b7b9e74baef4efec33e3c" Oct 30 18:24:01.689583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:01.689560 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:24:01.689666 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:01.689616 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:24:01.689666 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:01.689657 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:24:02.112561 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:02.112488 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:02.742527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:02.725115 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" event={"ID":"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7","Type":"ContainerStarted","Data":"c67379c924b541a3ea81649e0ec3507aa1ee9ffec376c32531fb98c7076bd5a3"} Oct 30 18:24:02.742527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:02.742409 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" event={"ID":"9d3f60a0-c196-48e7-9765-b1f3d8965367","Type":"ContainerStarted","Data":"8c43fa786feb1975bf4a240378fbd2b573782e07262b74fabca7f0a43454d83c"} Oct 30 18:24:02.851654 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:02.851627 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:24:02.852604 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:02.852584 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 18:24:02.852743 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:02.852626 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" podUID="e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 18:24:02.856574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:02.856547 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:03.063215 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:03.063184 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:24:03.063620 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:03.063590 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 18:24:03.063915 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:03.063824 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" podUID="9d3f60a0-c196-48e7-9765-b1f3d8965367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 18:24:03.839009 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:03.838980 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:03.862543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:03.858094 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 18:24:03.862543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:03.858218 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" podUID="e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 18:24:04.064528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:04.064319 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 18:24:04.064528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:04.064383 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" podUID="9d3f60a0-c196-48e7-9765-b1f3d8965367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 18:24:04.852227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:04.852194 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 18:24:04.852669 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:04.852254 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" podUID="e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 18:24:05.061997 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:05.061965 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 18:24:05.062188 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:05.062019 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" podUID="9d3f60a0-c196-48e7-9765-b1f3d8965367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 18:24:05.641527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:05.640871 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:05.856307 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:05.855885 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 18:24:05.856307 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:05.855940 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" podUID="e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 18:24:06.062109 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:06.062073 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 18:24:06.062304 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:06.062126 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" podUID="9d3f60a0-c196-48e7-9765-b1f3d8965367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 18:24:06.505684 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:06.505595 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:24:06.852492 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:06.852458 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 18:24:06.852682 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:06.852531 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" podUID="e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 18:24:06.953193 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:06.953163 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:07.061971 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:07.061939 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 18:24:07.062148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:07.061995 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" podUID="9d3f60a0-c196-48e7-9765-b1f3d8965367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 18:24:07.284070 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:07.284034 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:07.448860 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:07.447676 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:07.631583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:07.607829 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:07.851627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:07.851594 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 18:24:07.851926 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:07.851895 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" podUID="e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 18:24:08.064332 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:08.064295 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 18:24:08.064973 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:08.064940 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" podUID="9d3f60a0-c196-48e7-9765-b1f3d8965367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 18:24:08.874686 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:08.874657 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:24:08.877191 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:08.877166 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:24:09.051829 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:09.051799 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:09.066532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:09.066473 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:24:09.066886 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:09.066781 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:24:09.067439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:09.067423 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:24:09.555139 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:09.555106 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:09.710991 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:09.710960 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:10.985540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:10.973828 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:11.425748 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:11.425710 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:11.576837 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:11.576805 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:12.726771 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:12.726740 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:14.139541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:14.139485 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:14.417085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:14.417055 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:14.617134 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:14.617100 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:16.086877 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:16.086843 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:16.241495 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:16.241459 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:17.730910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:17.730867 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:19.479853 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:19.479814 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:21.162948 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:21.162912 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:21.933792 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:24:21.933748 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:24:22.754091 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:22.754051 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:24.241954 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:24.241919 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:24.440972 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:24.440943 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:24.570047 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:24.570005 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs"] Oct 30 18:24:24.578882 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:24.578854 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:24.594911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:24.594873 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-c2b103-openshift-default-dockercfg-2gsp2\"" Oct 30 18:24:24.823631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:24.721000 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr46s\" (UniqueName: \"kubernetes.io/projected/2546a5f3-a662-42f8-91a2-0870880c3291-kube-api-access-kr46s\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:24.877106 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:24.877070 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs"] Oct 30 18:24:24.943614 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:24.901826 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-workload-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:24.943936 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:24.943645 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2546a5f3-a662-42f8-91a2-0870880c3291-istiod-ca-cert\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:24.943936 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:24.943692 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2546a5f3-a662-42f8-91a2-0870880c3291-istio-podinfo\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:24.943936 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:24.943730 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-istio-envoy\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:24.943936 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:24.943784 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-workload-certs\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:24.943936 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:24.943888 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2546a5f3-a662-42f8-91a2-0870880c3291-istio-token\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:24.944195 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:24.943943 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-istio-data\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:24.944195 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:24.944017 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-credential-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:24.944327 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:24.939602 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:25.060785 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.045560 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-workload-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:25.087596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.061083 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2546a5f3-a662-42f8-91a2-0870880c3291-istiod-ca-cert\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:25.087978 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.087944 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2546a5f3-a662-42f8-91a2-0870880c3291-istio-podinfo\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:25.088182 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.088164 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-istio-envoy\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:25.088332 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.088317 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-workload-certs\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:25.088588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.088564 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2546a5f3-a662-42f8-91a2-0870880c3291-istio-token\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:25.088787 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.088772 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-istio-data\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:25.088958 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.088939 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-credential-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:25.089285 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.089262 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr46s\" (UniqueName: \"kubernetes.io/projected/2546a5f3-a662-42f8-91a2-0870880c3291-kube-api-access-kr46s\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:25.115526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.093879 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2546a5f3-a662-42f8-91a2-0870880c3291-istiod-ca-cert\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:25.115526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.094331 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-istio-data\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:25.115526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.094594 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-workload-certs\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:25.115526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.095356 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-credential-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:25.115526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.097860 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-workload-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:25.187694 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.187658 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:25.195097 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.189245 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2546a5f3-a662-42f8-91a2-0870880c3291-istio-podinfo\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:25.196048 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.196013 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2546a5f3-a662-42f8-91a2-0870880c3291-istio-token\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:25.206150 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.206117 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr46s\" (UniqueName: \"kubernetes.io/projected/2546a5f3-a662-42f8-91a2-0870880c3291-kube-api-access-kr46s\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:25.212610 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.212580 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-istio-envoy\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:25.507023 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.506944 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:25.836431 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.836403 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs"] Oct 30 18:24:25.837163 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:24:25.837137 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2546a5f3_a662_42f8_91a2_0870880c3291.slice/crio-3cecea1b55e81dc255845d3c48ddcb6a4532d92ca2e1925ecf20546d3224d890 WatchSource:0}: Error finding container 3cecea1b55e81dc255845d3c48ddcb6a4532d92ca2e1925ecf20546d3224d890: Status 404 returned error can't find the container with id 3cecea1b55e81dc255845d3c48ddcb6a4532d92ca2e1925ecf20546d3224d890 Oct 30 18:24:25.839926 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.839883 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:24:25.840013 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.839966 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:24:25.840013 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:25.839995 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:24:26.218494 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:26.218404 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" event={"ID":"2546a5f3-a662-42f8-91a2-0870880c3291","Type":"ContainerStarted","Data":"80167c0ce897f4f7e70aab6ec21b6448441828671060fbd79488eb94d2d07f16"} Oct 30 18:24:26.218494 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:26.218463 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" event={"ID":"2546a5f3-a662-42f8-91a2-0870880c3291","Type":"ContainerStarted","Data":"3cecea1b55e81dc255845d3c48ddcb6a4532d92ca2e1925ecf20546d3224d890"} Oct 30 18:24:26.289602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:26.289550 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" podStartSLOduration=2.289536329 podStartE2EDuration="2.289536329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:26.275071896 +0000 UTC m=+1926.990641782" watchObservedRunningTime="2026-10-30 18:24:26.289536329 +0000 UTC m=+1927.005106204" Oct 30 18:24:26.340128 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:26.340094 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:26.510214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:26.510115 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:26.523257 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:26.523228 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:27.225368 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:27.225341 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:27.226269 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:27.226245 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:24:28.167450 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:28.167396 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:28.690980 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:28.690926 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:28.858497 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:28.858463 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:30.111391 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:30.111361 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:30.366280 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:30.366112 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:30.509730 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:30.509697 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:32.055121 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:32.055085 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:32.821750 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:32.821718 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:32.970181 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:32.970153 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr"] Oct 30 18:24:32.974569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:32.974550 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr" Oct 30 18:24:32.980933 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:32.980909 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:32.999137 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:32.999112 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr"] Oct 30 18:24:33.003064 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:33.003045 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-c2b103\"" Oct 30 18:24:33.003162 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:33.003097 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-c2b103-dockercfg-6tj2g\"" Oct 30 18:24:33.399632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:33.396134 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46v68\" (UniqueName: \"kubernetes.io/projected/831276c5-adb1-40ab-8bd2-2aa6ec757865-kube-api-access-46v68\") pod \"payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr\" (UID: \"831276c5-adb1-40ab-8bd2-2aa6ec757865\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr" Oct 30 18:24:33.399632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:33.396328 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/831276c5-adb1-40ab-8bd2-2aa6ec757865-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr\" (UID: \"831276c5-adb1-40ab-8bd2-2aa6ec757865\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr" Oct 30 18:24:33.415898 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:33.415855 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m"] Oct 30 18:24:33.434769 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:33.434735 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m"] Oct 30 18:24:33.434940 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:33.434855 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m" Oct 30 18:24:33.497669 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:33.497633 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46v68\" (UniqueName: \"kubernetes.io/projected/831276c5-adb1-40ab-8bd2-2aa6ec757865-kube-api-access-46v68\") pod \"payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr\" (UID: \"831276c5-adb1-40ab-8bd2-2aa6ec757865\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr" Oct 30 18:24:33.497832 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:33.497759 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hlnd\" (UniqueName: \"kubernetes.io/projected/9641ab7d-afbe-405b-b4b6-f01f0d403c84-kube-api-access-8hlnd\") pod \"payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m\" (UID: \"9641ab7d-afbe-405b-b4b6-f01f0d403c84\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m" Oct 30 18:24:33.497832 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:33.497793 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/831276c5-adb1-40ab-8bd2-2aa6ec757865-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr\" (UID: \"831276c5-adb1-40ab-8bd2-2aa6ec757865\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr" Oct 30 18:24:33.497957 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:33.497936 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9641ab7d-afbe-405b-b4b6-f01f0d403c84-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m\" (UID: \"9641ab7d-afbe-405b-b4b6-f01f0d403c84\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m" Oct 30 18:24:33.498481 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:33.498456 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/831276c5-adb1-40ab-8bd2-2aa6ec757865-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr\" (UID: \"831276c5-adb1-40ab-8bd2-2aa6ec757865\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr" Oct 30 18:24:33.520981 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:33.520951 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46v68\" (UniqueName: \"kubernetes.io/projected/831276c5-adb1-40ab-8bd2-2aa6ec757865-kube-api-access-46v68\") pod \"payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr\" (UID: \"831276c5-adb1-40ab-8bd2-2aa6ec757865\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr" Oct 30 18:24:33.589949 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:33.589920 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr" Oct 30 18:24:33.598784 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:33.598762 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hlnd\" (UniqueName: \"kubernetes.io/projected/9641ab7d-afbe-405b-b4b6-f01f0d403c84-kube-api-access-8hlnd\") pod \"payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m\" (UID: \"9641ab7d-afbe-405b-b4b6-f01f0d403c84\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m" Oct 30 18:24:33.598901 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:33.598882 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9641ab7d-afbe-405b-b4b6-f01f0d403c84-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m\" (UID: \"9641ab7d-afbe-405b-b4b6-f01f0d403c84\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m" Oct 30 18:24:33.599781 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:33.599761 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9641ab7d-afbe-405b-b4b6-f01f0d403c84-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m\" (UID: \"9641ab7d-afbe-405b-b4b6-f01f0d403c84\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m" Oct 30 18:24:33.621891 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:33.621864 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hlnd\" (UniqueName: \"kubernetes.io/projected/9641ab7d-afbe-405b-b4b6-f01f0d403c84-kube-api-access-8hlnd\") pod \"payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m\" (UID: \"9641ab7d-afbe-405b-b4b6-f01f0d403c84\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m" Oct 30 18:24:33.754233 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:33.754182 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m" Oct 30 18:24:33.906974 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:24:33.906946 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod831276c5_adb1_40ab_8bd2_2aa6ec757865.slice/crio-b756216de18049b34733665bd6ff9dd34fe4c4c182a35c9dd84df44dd165c8a3 WatchSource:0}: Error finding container b756216de18049b34733665bd6ff9dd34fe4c4c182a35c9dd84df44dd165c8a3: Status 404 returned error can't find the container with id b756216de18049b34733665bd6ff9dd34fe4c4c182a35c9dd84df44dd165c8a3 Oct 30 18:24:33.910821 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:33.910794 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr"] Oct 30 18:24:33.954654 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:33.954624 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:34.108066 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:34.105651 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:34.128971 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:34.128809 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m"] Oct 30 18:24:34.135262 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:24:34.135214 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9641ab7d_afbe_405b_b4b6_f01f0d403c84.slice/crio-d8852686cf524f7c457d1a1eac8a81dbdd5038ed3466a6d85772c0a5b4d0a06d WatchSource:0}: Error finding container d8852686cf524f7c457d1a1eac8a81dbdd5038ed3466a6d85772c0a5b4d0a06d: Status 404 returned error can't find the container with id d8852686cf524f7c457d1a1eac8a81dbdd5038ed3466a6d85772c0a5b4d0a06d Oct 30 18:24:34.415494 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:34.415449 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr" event={"ID":"831276c5-adb1-40ab-8bd2-2aa6ec757865","Type":"ContainerStarted","Data":"1e831e77dc7f594d3e9dec2a36a7968760ee3e1a9d162ac4608d6ff22f3284a5"} Oct 30 18:24:34.415954 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:34.415536 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr" Oct 30 18:24:34.415954 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:34.415567 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr" event={"ID":"831276c5-adb1-40ab-8bd2-2aa6ec757865","Type":"ContainerStarted","Data":"b756216de18049b34733665bd6ff9dd34fe4c4c182a35c9dd84df44dd165c8a3"} Oct 30 18:24:34.417259 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:34.417232 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m" event={"ID":"9641ab7d-afbe-405b-b4b6-f01f0d403c84","Type":"ContainerStarted","Data":"5a6e86c15c20f6e68ba24fb50dc88d9fecedcac9385d4b27bd1ce19e1a64d157"} Oct 30 18:24:34.417365 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:34.417282 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m" event={"ID":"9641ab7d-afbe-405b-b4b6-f01f0d403c84","Type":"ContainerStarted","Data":"d8852686cf524f7c457d1a1eac8a81dbdd5038ed3466a6d85772c0a5b4d0a06d"} Oct 30 18:24:34.417365 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:34.417305 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m" Oct 30 18:24:34.458306 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:34.458231 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr" podStartSLOduration=2.4582194299999998 podStartE2EDuration="2.45821943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:34.455546089 +0000 UTC m=+1935.171115962" watchObservedRunningTime="2026-10-30 18:24:34.45821943 +0000 UTC m=+1935.173789307" Oct 30 18:24:34.494935 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:34.494887 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m" podStartSLOduration=1.494875852 podStartE2EDuration="1.494875852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:34.494245228 +0000 UTC m=+1935.209815101" watchObservedRunningTime="2026-10-30 18:24:34.494875852 +0000 UTC m=+1935.210445725" Oct 30 18:24:35.652865 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:35.652831 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:37.023412 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:37.023381 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:37.176464 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:37.176039 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:37.473470 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:37.473381 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:39.416423 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:39.416391 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:40.931060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:40.931023 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:41.153115 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:41.153083 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:42.160567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:42.160490 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:42.651924 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:42.651892 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:42.816357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:42.816325 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:44.477729 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:44.477688 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:45.427063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:45.427037 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr" Oct 30 18:24:45.427220 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:45.427100 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m" Oct 30 18:24:45.909321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:45.909283 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:46.464542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:46.463429 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:46.617209 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:46.617114 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:46.651211 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:46.651168 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85"] Oct 30 18:24:46.774533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:46.773611 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:46.946589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:46.939365 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:47.177717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:47.177624 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:48.753101 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:48.753074 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:48.907031 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:48.906995 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:49.693969 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:49.693933 4603 generic.go:356] "Generic (PLEG): container finished" podID="13daa509-1252-49e8-80ad-caeaedeec185" containerID="a7716bf32ed1ffe7b93edb296974cbf967ad8f0772a020f1001ce9320da2e4ce" exitCode=0 Oct 30 18:24:49.694180 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:49.694005 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" event={"ID":"13daa509-1252-49e8-80ad-caeaedeec185","Type":"ContainerDied","Data":"a7716bf32ed1ffe7b93edb296974cbf967ad8f0772a020f1001ce9320da2e4ce"} Oct 30 18:24:49.694483 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:49.694464 4603 scope.go:122] "RemoveContainer" containerID="a7716bf32ed1ffe7b93edb296974cbf967ad8f0772a020f1001ce9320da2e4ce" Oct 30 18:24:49.696074 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:49.696041 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:24:49.696177 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:49.696128 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:24:49.696247 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:49.696180 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:24:50.375578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:50.375541 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:50.540797 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:50.540720 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:50.682717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:50.682625 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:50.702758 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:50.702728 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" event={"ID":"13daa509-1252-49e8-80ad-caeaedeec185","Type":"ContainerStarted","Data":"6b1c93f4e4cb37e420d75a806dfe687329da51d76bc0e73a096702a03a589e11"} Oct 30 18:24:51.168488 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:51.168457 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:24:51.169456 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:51.169431 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 18:24:51.169598 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:51.169490 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" podUID="13daa509-1252-49e8-80ad-caeaedeec185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 18:24:51.652390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:51.652348 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85" podUID="fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef" containerName="payload-processing" containerID="cri-o://45ba5ac3525f19ce149c17114894b57d932897a6a433b46ca9256af4c37ad121" gracePeriod=25 Oct 30 18:24:51.956555 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:24:51.956485 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:24:51.985777 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:51.985753 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85" Oct 30 18:24:52.037003 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:52.036969 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:52.191931 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:52.191894 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef-kube-api-access-wkt6n\" (UniqueName: \"kubernetes.io/projected/fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef-kube-api-access-wkt6n\") pod \"fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef\" (UID: \"fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef\") " Oct 30 18:24:52.204530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:52.192396 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef-plugins-config-volume\") pod \"fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef\" (UID: \"fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef\") " Oct 30 18:24:52.231424 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:52.231339 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 18:24:52.231712 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:52.231684 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" podUID="13daa509-1252-49e8-80ad-caeaedeec185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 18:24:52.233175 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:52.233150 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef-plugins-config-volume" pod "fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef" (UID: "fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:52.252450 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:52.252415 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef-kube-api-access-wkt6n" pod "fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef" (UID: "fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef"). InnerVolumeSpecName "kube-api-access-wkt6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:52.319527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:52.319382 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkt6n\" (UniqueName: \"kubernetes.io/projected/fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef-kube-api-access-wkt6n\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:24:52.319527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:52.319419 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:24:52.753646 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:52.753604 4603 generic.go:356] "Generic (PLEG): container finished" podID="fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef" containerID="45ba5ac3525f19ce149c17114894b57d932897a6a433b46ca9256af4c37ad121" exitCode=0 Oct 30 18:24:52.754121 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:52.753646 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85" event={"ID":"fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef","Type":"ContainerDied","Data":"45ba5ac3525f19ce149c17114894b57d932897a6a433b46ca9256af4c37ad121"} Oct 30 18:24:52.754121 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:52.753658 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85" Oct 30 18:24:52.754121 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:52.753711 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85" event={"ID":"fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef","Type":"ContainerDied","Data":"ba2c0d3179a0fc70ea819e058683f3cebeb351b816e135faf8973c6502c33a59"} Oct 30 18:24:52.754121 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:52.753740 4603 scope.go:122] "RemoveContainer" containerID="45ba5ac3525f19ce149c17114894b57d932897a6a433b46ca9256af4c37ad121" Oct 30 18:24:52.764767 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:52.764747 4603 scope.go:122] "RemoveContainer" containerID="45ba5ac3525f19ce149c17114894b57d932897a6a433b46ca9256af4c37ad121" Oct 30 18:24:52.765040 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:24:52.765022 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45ba5ac3525f19ce149c17114894b57d932897a6a433b46ca9256af4c37ad121\": container with ID starting with 45ba5ac3525f19ce149c17114894b57d932897a6a433b46ca9256af4c37ad121 not found: ID does not exist" containerID="45ba5ac3525f19ce149c17114894b57d932897a6a433b46ca9256af4c37ad121" Oct 30 18:24:52.765104 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:52.765048 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45ba5ac3525f19ce149c17114894b57d932897a6a433b46ca9256af4c37ad121"} err="failed to get container status \"45ba5ac3525f19ce149c17114894b57d932897a6a433b46ca9256af4c37ad121\": rpc error: code = NotFound desc = could not find container \"45ba5ac3525f19ce149c17114894b57d932897a6a433b46ca9256af4c37ad121\": container with ID starting with 45ba5ac3525f19ce149c17114894b57d932897a6a433b46ca9256af4c37ad121 not found: ID does not exist" Oct 30 18:24:52.816838 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:52.816801 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85"] Oct 30 18:24:52.821561 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:52.821529 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d42172-9c865cb64-pbn85"] Oct 30 18:24:53.114966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:53.114935 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:53.171146 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:53.171103 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 18:24:53.171370 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:53.171188 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" podUID="13daa509-1252-49e8-80ad-caeaedeec185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 18:24:53.261832 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:53.261796 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:53.365593 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:53.365471 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss"] Oct 30 18:24:53.365817 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:53.365757 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" podUID="13daa509-1252-49e8-80ad-caeaedeec185" containerName="istio-proxy" containerID="cri-o://6b1c93f4e4cb37e420d75a806dfe687329da51d76bc0e73a096702a03a589e11" gracePeriod=30 Oct 30 18:24:53.432632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:53.432592 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:53.433081 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:53.433061 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef" path="/var/lib/kubelet/pods/fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef/volumes" Oct 30 18:24:53.850532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:53.839573 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:54.847327 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:54.847296 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:24:55.753027 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:55.752989 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:56.712524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:56.712359 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:56.866445 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:56.866401 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:57.024084 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:57.024046 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:57.351638 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:57.351564 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:58.017456 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:58.017419 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:58.292376 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:58.292341 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:58.446750 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:58.446720 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:58.601355 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:58.601254 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:58.755736 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:58.755703 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:59.128528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:24:59.092071 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:00.236226 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:00.236184 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:02.056039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.056006 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:02.348831 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.348809 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:25:02.437518 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.437465 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-istio-data\") pod \"13daa509-1252-49e8-80ad-caeaedeec185\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " Oct 30 18:25:02.437689 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.437590 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13daa509-1252-49e8-80ad-caeaedeec185-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13daa509-1252-49e8-80ad-caeaedeec185-istio-podinfo\") pod \"13daa509-1252-49e8-80ad-caeaedeec185\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " Oct 30 18:25:02.437689 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.437678 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13daa509-1252-49e8-80ad-caeaedeec185-kube-api-access-tkjkx\" (UniqueName: \"kubernetes.io/projected/13daa509-1252-49e8-80ad-caeaedeec185-kube-api-access-tkjkx\") pod \"13daa509-1252-49e8-80ad-caeaedeec185\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " Oct 30 18:25:02.437843 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.437735 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-credential-socket\") pod \"13daa509-1252-49e8-80ad-caeaedeec185\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " Oct 30 18:25:02.437906 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.437857 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-istio-envoy\") pod \"13daa509-1252-49e8-80ad-caeaedeec185\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " Oct 30 18:25:02.437906 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.437900 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13daa509-1252-49e8-80ad-caeaedeec185-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13daa509-1252-49e8-80ad-caeaedeec185-istiod-ca-cert\") pod \"13daa509-1252-49e8-80ad-caeaedeec185\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " Oct 30 18:25:02.438030 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.437948 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-workload-socket\") pod \"13daa509-1252-49e8-80ad-caeaedeec185\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " Oct 30 18:25:02.438030 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.437993 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-workload-certs\") pod \"13daa509-1252-49e8-80ad-caeaedeec185\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " Oct 30 18:25:02.438158 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.438056 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13daa509-1252-49e8-80ad-caeaedeec185-istio-token\" (UniqueName: \"kubernetes.io/projected/13daa509-1252-49e8-80ad-caeaedeec185-istio-token\") pod \"13daa509-1252-49e8-80ad-caeaedeec185\" (UID: \"13daa509-1252-49e8-80ad-caeaedeec185\") " Oct 30 18:25:02.438337 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.438255 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-credential-socket" pod "13daa509-1252-49e8-80ad-caeaedeec185" (UID: "13daa509-1252-49e8-80ad-caeaedeec185"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.438906 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.438637 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-workload-socket" pod "13daa509-1252-49e8-80ad-caeaedeec185" (UID: "13daa509-1252-49e8-80ad-caeaedeec185"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.438906 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.438847 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13daa509-1252-49e8-80ad-caeaedeec185-istiod-ca-cert" pod "13daa509-1252-49e8-80ad-caeaedeec185" (UID: "13daa509-1252-49e8-80ad-caeaedeec185"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:02.439077 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.438988 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.439077 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.439008 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.439880 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.439842 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-istio-data" pod "13daa509-1252-49e8-80ad-caeaedeec185" (UID: "13daa509-1252-49e8-80ad-caeaedeec185"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.440059 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.440035 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-workload-certs" pod "13daa509-1252-49e8-80ad-caeaedeec185" (UID: "13daa509-1252-49e8-80ad-caeaedeec185"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.441823 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.441796 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13daa509-1252-49e8-80ad-caeaedeec185-kube-api-access-tkjkx" pod "13daa509-1252-49e8-80ad-caeaedeec185" (UID: "13daa509-1252-49e8-80ad-caeaedeec185"). InnerVolumeSpecName "kube-api-access-tkjkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:02.442103 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.442076 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13daa509-1252-49e8-80ad-caeaedeec185-istio-podinfo" pod "13daa509-1252-49e8-80ad-caeaedeec185" (UID: "13daa509-1252-49e8-80ad-caeaedeec185"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:02.442420 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.442398 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-istio-envoy" pod "13daa509-1252-49e8-80ad-caeaedeec185" (UID: "13daa509-1252-49e8-80ad-caeaedeec185"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.443977 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.443952 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13daa509-1252-49e8-80ad-caeaedeec185-istio-token" pod "13daa509-1252-49e8-80ad-caeaedeec185" (UID: "13daa509-1252-49e8-80ad-caeaedeec185"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:02.540148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.540119 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.540148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.540144 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13daa509-1252-49e8-80ad-caeaedeec185-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.540148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.540152 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.540324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.540160 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13daa509-1252-49e8-80ad-caeaedeec185-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.540324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.540169 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13daa509-1252-49e8-80ad-caeaedeec185-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.540324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.540177 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13daa509-1252-49e8-80ad-caeaedeec185-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.540324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:02.540186 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkjkx\" (UniqueName: \"kubernetes.io/projected/13daa509-1252-49e8-80ad-caeaedeec185-kube-api-access-tkjkx\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:03.153493 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:03.153457 4603 generic.go:356] "Generic (PLEG): container finished" podID="13daa509-1252-49e8-80ad-caeaedeec185" containerID="6b1c93f4e4cb37e420d75a806dfe687329da51d76bc0e73a096702a03a589e11" exitCode=0 Oct 30 18:25:03.154019 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:03.153519 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" Oct 30 18:25:03.154019 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:03.153558 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" event={"ID":"13daa509-1252-49e8-80ad-caeaedeec185","Type":"ContainerDied","Data":"6b1c93f4e4cb37e420d75a806dfe687329da51d76bc0e73a096702a03a589e11"} Oct 30 18:25:03.154019 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:03.153637 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss" event={"ID":"13daa509-1252-49e8-80ad-caeaedeec185","Type":"ContainerDied","Data":"2afccea54ff7870e1e8d9a0c0eff5ac1a6de2878d8d28d38e69b70e97c9ef247"} Oct 30 18:25:03.154019 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:03.153661 4603 scope.go:122] "RemoveContainer" containerID="6b1c93f4e4cb37e420d75a806dfe687329da51d76bc0e73a096702a03a589e11" Oct 30 18:25:03.164495 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:03.164299 4603 scope.go:122] "RemoveContainer" containerID="a7716bf32ed1ffe7b93edb296974cbf967ad8f0772a020f1001ce9320da2e4ce" Oct 30 18:25:03.174157 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:03.174142 4603 scope.go:122] "RemoveContainer" containerID="6b1c93f4e4cb37e420d75a806dfe687329da51d76bc0e73a096702a03a589e11" Oct 30 18:25:03.174397 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:25:03.174378 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b1c93f4e4cb37e420d75a806dfe687329da51d76bc0e73a096702a03a589e11\": container with ID starting with 6b1c93f4e4cb37e420d75a806dfe687329da51d76bc0e73a096702a03a589e11 not found: ID does not exist" containerID="6b1c93f4e4cb37e420d75a806dfe687329da51d76bc0e73a096702a03a589e11" Oct 30 18:25:03.174450 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:03.174406 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b1c93f4e4cb37e420d75a806dfe687329da51d76bc0e73a096702a03a589e11"} err="failed to get container status \"6b1c93f4e4cb37e420d75a806dfe687329da51d76bc0e73a096702a03a589e11\": rpc error: code = NotFound desc = could not find container \"6b1c93f4e4cb37e420d75a806dfe687329da51d76bc0e73a096702a03a589e11\": container with ID starting with 6b1c93f4e4cb37e420d75a806dfe687329da51d76bc0e73a096702a03a589e11 not found: ID does not exist" Oct 30 18:25:03.174450 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:03.174424 4603 scope.go:122] "RemoveContainer" containerID="a7716bf32ed1ffe7b93edb296974cbf967ad8f0772a020f1001ce9320da2e4ce" Oct 30 18:25:03.174695 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:25:03.174676 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7716bf32ed1ffe7b93edb296974cbf967ad8f0772a020f1001ce9320da2e4ce\": container with ID starting with a7716bf32ed1ffe7b93edb296974cbf967ad8f0772a020f1001ce9320da2e4ce not found: ID does not exist" containerID="a7716bf32ed1ffe7b93edb296974cbf967ad8f0772a020f1001ce9320da2e4ce" Oct 30 18:25:03.174774 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:03.174705 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7716bf32ed1ffe7b93edb296974cbf967ad8f0772a020f1001ce9320da2e4ce"} err="failed to get container status \"a7716bf32ed1ffe7b93edb296974cbf967ad8f0772a020f1001ce9320da2e4ce\": rpc error: code = NotFound desc = could not find container \"a7716bf32ed1ffe7b93edb296974cbf967ad8f0772a020f1001ce9320da2e4ce\": container with ID starting with a7716bf32ed1ffe7b93edb296974cbf967ad8f0772a020f1001ce9320da2e4ce not found: ID does not exist" Oct 30 18:25:03.222318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:03.222291 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss"] Oct 30 18:25:03.234160 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:03.234136 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-ef208a-openshift-default-6594775cd5-7glss"] Oct 30 18:25:03.411578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:03.411514 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13daa509-1252-49e8-80ad-caeaedeec185" path="/var/lib/kubelet/pods/13daa509-1252-49e8-80ad-caeaedeec185/volumes" Oct 30 18:25:03.730351 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:03.730266 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:04.550769 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:04.550742 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:04.703611 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:04.703582 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:04.889581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:04.866128 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:05.347623 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:05.347587 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:05.490020 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:05.489989 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:05.780567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:05.780535 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q"] Oct 30 18:25:05.781222 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:05.781191 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" podUID="e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" containerName="istio-proxy" containerID="cri-o://c67379c924b541a3ea81649e0ec3507aa1ee9ffec376c32531fb98c7076bd5a3" gracePeriod=30 Oct 30 18:25:05.783183 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:05.783158 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:07.143568 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:07.143534 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:08.770075 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:08.770033 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:25:08.913612 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:08.913575 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:25:08.913950 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:08.913919 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" podUID="e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:25:08.944529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:08.943410 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:10.680644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:10.680608 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:12.238102 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.234497 4603 generic.go:356] "Generic (PLEG): container finished" podID="e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" containerID="c67379c924b541a3ea81649e0ec3507aa1ee9ffec376c32531fb98c7076bd5a3" exitCode=0 Oct 30 18:25:12.284745 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.284710 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:25:12.446636 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.446585 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:25:12.636529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.502317 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:12.704787 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.704742 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:12.712530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.708812 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:25:12.751526 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:25:12.745262 4603 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.102s" Oct 30 18:25:12.751526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.745420 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" event={"ID":"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7","Type":"ContainerDied","Data":"c67379c924b541a3ea81649e0ec3507aa1ee9ffec376c32531fb98c7076bd5a3"} Oct 30 18:25:12.806524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.789628 4603 scope.go:122] "RemoveContainer" containerID="c67379c924b541a3ea81649e0ec3507aa1ee9ffec376c32531fb98c7076bd5a3" Oct 30 18:25:12.838523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.837523 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-kube-api-access-8zxcl\" (UniqueName: \"kubernetes.io/projected/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-kube-api-access-8zxcl\") pod \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " Oct 30 18:25:12.838523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.837627 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-credential-socket\") pod \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " Oct 30 18:25:12.838523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.837753 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-envoy\") pod \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " Oct 30 18:25:12.838523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.837797 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-workload-certs\") pod \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " Oct 30 18:25:12.838523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.837923 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-token\" (UniqueName: \"kubernetes.io/projected/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-token\") pod \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " Oct 30 18:25:12.838523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.837963 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-podinfo\") pod \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " Oct 30 18:25:12.838523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.838143 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istiod-ca-cert\") pod \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " Oct 30 18:25:12.838523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.838187 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-workload-socket\") pod \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " Oct 30 18:25:12.838523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.838270 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-data\") pod \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\" (UID: \"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7\") " Oct 30 18:25:12.871536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.840418 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-data" pod "e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" (UID: "e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:12.871536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.869287 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-workload-socket" pod "e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" (UID: "e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:12.871536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.870536 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-workload-certs" pod "e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" (UID: "e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:12.872195 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.872168 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-credential-socket" pod "e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" (UID: "e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:12.872465 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.872430 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istiod-ca-cert" pod "e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" (UID: "e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:12.913566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.911748 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-kube-api-access-8zxcl" pod "e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" (UID: "e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7"). InnerVolumeSpecName "kube-api-access-8zxcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:12.931201 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.914663 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-podinfo" pod "e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" (UID: "e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:12.938566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.931911 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-envoy" pod "e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" (UID: "e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:12.945589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.941714 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-token" pod "e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" (UID: "e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:12.945589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.942769 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:12.945589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.942797 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:12.945589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.942810 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:12.945589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.942823 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zxcl\" (UniqueName: \"kubernetes.io/projected/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-kube-api-access-8zxcl\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:12.945589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.942836 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:12.945589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.942848 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:12.945589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.942860 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:12.945589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.942873 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:12.945589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:12.942884 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:13.022570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:13.012869 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:13.024600 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:13.023227 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:13.026292 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:13.026269 4603 scope.go:122] "RemoveContainer" containerID="f63162a2d6737b4440315e5f1da156b440d6a38db7e14306151fbc7d44df959a" Oct 30 18:25:13.093005 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:13.090669 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:13.228765 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:13.228656 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:13.314783 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:13.314734 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" event={"ID":"e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7","Type":"ContainerDied","Data":"f53a02072773cfea7eda4aee8c81595211e40f416bb8caa9710f8e532aa3b181"} Oct 30 18:25:13.315412 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:13.315390 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q" Oct 30 18:25:13.393543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:13.391663 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:13.393543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:13.391727 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" podUID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:13.417738 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:13.413902 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q"] Oct 30 18:25:13.486588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:13.475862 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:13.488303 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:13.488247 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-d42172-openshift-default-78d859f857-fx24q"] Oct 30 18:25:13.638717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:13.635925 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:25:13.638717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:13.636336 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:13.642546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:13.641724 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:25:13.642546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:13.641788 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:25:13.770401 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:13.766743 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:13.770401 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:13.767159 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:13.770401 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:13.768700 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:25:13.770401 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:13.768758 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:25:14.052805 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:14.051609 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:25:14.052805 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:14.052012 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:25:14.052805 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:14.052051 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:25:14.052805 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:14.052115 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:25:14.185234 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:14.136590 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:25:14.185234 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:14.137129 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:25:14.237635 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:14.235194 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:25:14.248545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:14.243976 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:25:14.607857 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:14.607780 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Oct 30 18:25:14.659547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:14.639201 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:14.693273 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:14.693230 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:14.773000 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:14.772958 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:25:14.773474 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:14.773260 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:25:14.947804 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:14.947704 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:14.948086 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:14.948052 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" podUID="81f49829-73ed-4760-9444-f1793861eba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:15.072530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:15.057643 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:15.072530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:15.058032 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:25:15.245588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:15.137789 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:25:15.269528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:15.251014 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:25:15.690545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:15.628303 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Oct 30 18:25:15.690545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:15.650599 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:25:15.811658 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:15.811609 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:25:15.861878 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:15.861834 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:15.935326 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:15.918445 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:25:16.058864 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:16.058821 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:16.060278 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:16.060246 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:25:16.157596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:16.157310 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" path="/var/lib/kubelet/pods/e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7/volumes" Oct 30 18:25:16.283085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:16.283039 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:25:16.283763 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:16.283736 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:25:16.666544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:16.665428 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:16.713083 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:16.713046 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:16.914530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:16.875558 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:16.914530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:16.913664 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:16.948725 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:16.947627 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:17.074528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:17.060607 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:17.074528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:17.061059 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:25:17.171680 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:17.171631 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:17.291772 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:17.291706 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:17.292578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:17.292539 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:25:17.340709 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:17.340651 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:17.671232 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:17.671131 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:17.885559 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:17.885488 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:25:17.920652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:17.920604 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:17.951030 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:17.950935 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.061720 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:18.061661 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:25:18.062194 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:18.062162 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.176846 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:18.176605 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.215380 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:18.215206 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:18.322740 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:18.322695 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.326896 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:18.326845 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:25:18.449402 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:18.449363 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:18.470748 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:18.457350 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 30 18:25:18.598527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:18.577623 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:18.598527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:18.577687 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" podUID="ec3c95aa-215c-4f15-8dc8-bc0beb954236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.723293 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:18.723195 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.724533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:18.723843 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Oct 30 18:25:18.912126 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:18.912078 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:25:18.925833 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:18.925793 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.956987 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:18.954824 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:18.956987 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:18.954885 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" podUID="b664f4ca-736e-4cb1-991d-985d1e6aff34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.956987 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:18.955324 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:25:19.043154 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:19.042752 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:19.043154 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:19.042820 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" podUID="4d6cd466-9f15-476d-9f04-6d03176f025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.084529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:19.063960 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.085147 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:19.081623 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:25:19.189541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:19.176786 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:19.189541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:19.176850 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" podUID="d4a3adf6-fb23-4334-8bab-f800b4e8e089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.345590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:19.297905 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:19.345590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:19.297982 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" podUID="1d187b51-f5c6-4a63-8300-2f01184a15d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.345590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:19.323599 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.345590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:19.335945 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.346195 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:19.346167 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:19.346390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:19.346368 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" podUID="b6e43478-8cc1-4452-8bc8-fed14f127896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.740904 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:19.740808 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.967530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:19.941604 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.967530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:19.942013 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:25:19.993632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:19.968707 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:20.086654 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:20.086601 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:20.166156 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:20.166116 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:25:20.337606 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:20.310754 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:20.337606 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:20.334640 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:25:20.343630 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:20.343584 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:20.744587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:20.743605 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:20.906827 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:20.906788 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:20.943632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:20.943198 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:21.004853 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:21.003536 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:21.101853 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:21.101803 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:21.439742 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:21.284915 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:25:21.440650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:21.343436 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:21.441103 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:21.345217 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:21.550693 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:21.550646 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:21.750129 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:21.749998 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:21.949781 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:21.949718 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:22.006711 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:22.006611 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:22.402876 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:25:22.402822 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:25:22.466010 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:22.465968 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:22.466460 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:22.466435 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:22.467143 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:22.466370 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:22.468326 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:22.466744 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:25:22.632584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:22.626628 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:25:22.746689 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:22.743463 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:22.795531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:22.776847 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:25:22.971610 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:22.954751 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:22.971610 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:22.954863 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:25:22.989550 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:22.984643 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:22.989550 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:22.984706 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" podUID="d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:23.018688 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:23.018629 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:25:23.107676 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:23.107638 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:23.107924 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:23.107898 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" podUID="f7882513-ef6d-4b11-871e-39a29ad022df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:23.341529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:23.340712 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:23.341529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:23.340803 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" podUID="58e2469b-dc0d-40ab-bd74-a47c4a9e438b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:23.481527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:23.480748 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:25:23.485523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:23.481748 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:25:23.485523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:23.482052 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:23.485523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:23.483343 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:25:23.627820 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:23.626798 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:23.802731 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:23.800882 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:25:23.958589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:23.956647 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:24.021182 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:24.021132 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:24.482539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:24.453222 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:24.484445 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:24.484401 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:24.485043 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:24.485015 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:25:24.485480 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:24.485455 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:25:24.486066 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:24.485904 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:25:24.725525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:24.691123 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:24.767601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:24.737064 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Oct 30 18:25:24.808689 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:24.808648 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:25:24.963713 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:24.963671 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:25.069637 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:25.022789 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:25.489354 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:25.489250 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:25:25.518374 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:25.489920 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:25:25.542614 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:25.490206 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:25:25.544243 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:25.490431 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:25:25.579067 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:25.577917 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:25.701555 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:25.692209 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:25.826841 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:25.826796 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:26.029767 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:26.029719 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:26.048624 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:26.033343 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:26.347174 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:26.347139 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:26.517839 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:26.517768 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:26.550728 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:26.550633 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:26.576545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:26.575036 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:26.577281 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:26.575074 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:26.701587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:26.700603 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:26.871529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:26.863607 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:25:27.057538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:27.049593 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:27.057538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:27.050229 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:27.136548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:27.133589 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Oct 30 18:25:27.368583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:27.364241 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded" Oct 30 18:25:27.520629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:27.520583 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:27.591285 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:27.591244 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:25:27.592579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:27.591576 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:27.605665 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:27.605226 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:25:27.708696 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:27.708595 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:27.865734 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:27.865677 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:28.061665 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:28.061620 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:28.062313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:28.062276 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:25:28.082665 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:28.082587 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:28.160855 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:28.155680 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:28.439848 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:28.439746 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 30 18:25:28.534531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:28.526696 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:28.600531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:28.593808 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:25:28.600531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:28.594417 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:28.618522 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:28.618464 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:28.619102 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:28.618575 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Oct 30 18:25:28.710098 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:28.709989 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:28.869531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:28.866585 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:29.067713 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:29.067647 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:29.069147 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:29.069118 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:29.529699 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:29.528627 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:29.600635 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:29.600589 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:29.601243 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:29.601207 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:25:29.636046 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:29.636004 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:29.724747 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:29.724684 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:25:29.881881 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:29.881773 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:30.062561 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:30.061303 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:30.070646 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:30.068627 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:30.070646 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:30.069537 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:25:30.289685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:30.289637 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:30.540715 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:30.540604 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:30.621652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:30.621605 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:30.622743 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:30.621881 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:30.646531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:30.643220 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:30.741525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:30.741205 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:25:30.899527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:30.882829 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:31.023546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:31.022715 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:25:31.069158 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:31.069067 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:25:31.070068 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:31.070028 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:31.402370 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:31.402286 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:25:31.500033 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:31.499022 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:25:31.500033 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:31.499080 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:25:31.500033 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:31.499127 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:25:31.525914 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:31.525885 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:25:31.537574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:31.530432 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:25:31.606765 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:31.606341 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:25:31.627485 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:31.627308 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:32.398362 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:32.393336 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:32.542256 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:32.542221 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:32.996614 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:32.996580 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:33.535960 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:33.531914 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:33.535960 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:33.532061 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" podUID="3e865ee8-6656-40a4-beca-34d6e43d4b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:33.535960 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:33.531859 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:33.535960 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:33.532793 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" podUID="fa0c0084-b530-436d-936a-b94ebc16fe03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:34.339706 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:34.339638 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:35.855022 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:35.854648 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:35.855022 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:35.854720 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:36.067523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:36.067316 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:37.842627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:37.842591 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:39.379001 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.378962 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7"] Oct 30 18:25:39.409063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.379720 4603 cpu_manager.go:414] "containerMap: removing container" podUID="e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" containerName="istio-proxy" Oct 30 18:25:39.409063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.379735 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" containerName="istio-proxy" Oct 30 18:25:39.409063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.379752 4603 cpu_manager.go:414] "containerMap: removing container" podUID="e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" containerName="istio-proxy" Oct 30 18:25:39.409063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.379757 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" containerName="istio-proxy" Oct 30 18:25:39.409063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.379780 4603 cpu_manager.go:414] "containerMap: removing container" podUID="13daa509-1252-49e8-80ad-caeaedeec185" containerName="istio-proxy" Oct 30 18:25:39.409063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.379785 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13daa509-1252-49e8-80ad-caeaedeec185" containerName="istio-proxy" Oct 30 18:25:39.409063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.379798 4603 cpu_manager.go:414] "containerMap: removing container" podUID="13daa509-1252-49e8-80ad-caeaedeec185" containerName="istio-proxy" Oct 30 18:25:39.409063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.379803 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13daa509-1252-49e8-80ad-caeaedeec185" containerName="istio-proxy" Oct 30 18:25:39.409063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.379809 4603 cpu_manager.go:414] "containerMap: removing container" podUID="fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef" containerName="payload-processing" Oct 30 18:25:39.409063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.379814 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef" containerName="payload-processing" Oct 30 18:25:39.409063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.379919 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="13daa509-1252-49e8-80ad-caeaedeec185" containerName="istio-proxy" Oct 30 18:25:39.409063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.379927 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd53dcf5-bf97-41b1-bb6b-96b9bb5383ef" containerName="payload-processing" Oct 30 18:25:39.409063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.379939 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" containerName="istio-proxy" Oct 30 18:25:39.409063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.379945 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c7cfe9-00f8-4962-a68f-ec84a2ae84d7" containerName="istio-proxy" Oct 30 18:25:39.409063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.379955 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="13daa509-1252-49e8-80ad-caeaedeec185" containerName="istio-proxy" Oct 30 18:25:39.419655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.419632 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:39.432394 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.432367 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7"] Oct 30 18:25:39.560136 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.560103 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c718da34-63db-4d45-84e9-bdc0e45ad26b-istiod-ca-cert\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:39.560432 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.560405 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-podinfo\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:39.560598 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.560460 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-token\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:39.560598 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.560566 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-credential-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:39.560742 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.560643 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-workload-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:39.560742 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.560688 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-workload-certs\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:39.560839 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.560776 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-data\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:39.560902 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.560878 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-envoy\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:39.560966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.560946 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cxhs\" (UniqueName: \"kubernetes.io/projected/c718da34-63db-4d45-84e9-bdc0e45ad26b-kube-api-access-6cxhs\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:39.614594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.605608 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:39.614594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.605647 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" podUID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.642565 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.662146 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-podinfo\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.662188 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-token\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.662225 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-credential-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.662245 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-workload-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.662282 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-workload-certs\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.662350 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-data\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.662391 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-envoy\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.662435 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxhs\" (UniqueName: \"kubernetes.io/projected/c718da34-63db-4d45-84e9-bdc0e45ad26b-kube-api-access-6cxhs\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.662558 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c718da34-63db-4d45-84e9-bdc0e45ad26b-istiod-ca-cert\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.662718 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-credential-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.662737 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-workload-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.662781 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-workload-certs\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.662794 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-data\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.663226 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c718da34-63db-4d45-84e9-bdc0e45ad26b-istiod-ca-cert\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.664786 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-podinfo\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.664792 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-envoy\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.684900 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cxhs\" (UniqueName: \"kubernetes.io/projected/c718da34-63db-4d45-84e9-bdc0e45ad26b-kube-api-access-6cxhs\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.685041 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-token\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.771662 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.823689 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.914720 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps"] Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.954573 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:39.965032 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps"] Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.005999 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl"] Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.006179 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" podUID="3e865ee8-6656-40a4-beca-34d6e43d4b8c" containerName="istio-proxy" containerID="cri-o://927bde0819ecda487e1d05e1c7ec9ecb1f29a963e8991c98d10f681171bb61e1" gracePeriod=30 Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.011435 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7"] Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.058368 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.075279 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xkw4\" (UniqueName: \"kubernetes.io/projected/933bdf35-df63-46f1-9cb5-62d484a4ecb0-kube-api-access-6xkw4\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.075403 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istiod-ca-cert\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.075469 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-data\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.075490 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-envoy\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.075661 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-credential-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.075737 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-workload-certs\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.075782 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-podinfo\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.075843 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-workload-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.075942 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-token\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.179529 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-workload-certs\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.179583 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-podinfo\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.179662 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-workload-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.179769 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-token\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.179840 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xkw4\" (UniqueName: \"kubernetes.io/projected/933bdf35-df63-46f1-9cb5-62d484a4ecb0-kube-api-access-6xkw4\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.179951 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istiod-ca-cert\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.180039 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-data\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.180075 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-envoy\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.180129 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-credential-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.180626 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-credential-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.182965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.180888 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-workload-certs\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.235336 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.189217 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-workload-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.235336 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.189772 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-data\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.235336 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.190439 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istiod-ca-cert\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.235336 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.208670 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-envoy\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.374970 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.255182 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xkw4\" (UniqueName: \"kubernetes.io/projected/933bdf35-df63-46f1-9cb5-62d484a4ecb0-kube-api-access-6xkw4\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.374970 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.258379 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-podinfo\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.374970 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.288128 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-token\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.374970 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.308933 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:40.421202 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.421172 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7"] Oct 30 18:25:40.425427 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:25:40.425397 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc718da34_63db_4d45_84e9_bdc0e45ad26b.slice/crio-206fd4141b83910d3a36bfaa2e3039ca34ff3119673cbd875b05ee6eea70f453 WatchSource:0}: Error finding container 206fd4141b83910d3a36bfaa2e3039ca34ff3119673cbd875b05ee6eea70f453: Status 404 returned error can't find the container with id 206fd4141b83910d3a36bfaa2e3039ca34ff3119673cbd875b05ee6eea70f453 Oct 30 18:25:40.430307 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.430266 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:40.430420 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.430349 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:40.430420 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.430392 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:40.716042 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.716014 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx"] Oct 30 18:25:40.734885 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.734007 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.741853 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.741811 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx"] Oct 30 18:25:40.753043 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.752949 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps"] Oct 30 18:25:40.776637 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.776600 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:40.776734 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.776694 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:40.776786 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.776742 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:40.781223 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.781197 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:40.798313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.798289 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.798431 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.798405 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30517af-31fe-458a-8b17-40bc5ab61274-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.798496 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.798466 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.798579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.798550 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.798644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.798593 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8jdz\" (UniqueName: \"kubernetes.io/projected/a30517af-31fe-458a-8b17-40bc5ab61274-kube-api-access-w8jdz\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.798644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.798630 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30517af-31fe-458a-8b17-40bc5ab61274-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.798774 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.798759 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.798833 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.798785 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30517af-31fe-458a-8b17-40bc5ab61274-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.798894 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.798861 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.830388 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.830362 4603 generic.go:356] "Generic (PLEG): container finished" podID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerID="c78b43d96281f3f140142fdc01deb094700395b4c1b15ddc1bc01e2b05f70932" exitCode=0 Oct 30 18:25:40.830524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.830437 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" event={"ID":"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af","Type":"ContainerDied","Data":"c78b43d96281f3f140142fdc01deb094700395b4c1b15ddc1bc01e2b05f70932"} Oct 30 18:25:40.830919 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.830893 4603 scope.go:122] "RemoveContainer" containerID="c78b43d96281f3f140142fdc01deb094700395b4c1b15ddc1bc01e2b05f70932" Oct 30 18:25:40.832328 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.832309 4603 generic.go:356] "Generic (PLEG): container finished" podID="fa0c0084-b530-436d-936a-b94ebc16fe03" containerID="bf290b19a878dbf3b3491b1985b16ba5030b0285dfeaf04cdee0238993694388" exitCode=0 Oct 30 18:25:40.832398 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.832382 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" event={"ID":"fa0c0084-b530-436d-936a-b94ebc16fe03","Type":"ContainerDied","Data":"bf290b19a878dbf3b3491b1985b16ba5030b0285dfeaf04cdee0238993694388"} Oct 30 18:25:40.832528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.832486 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:40.832587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.832572 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:40.832637 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.832610 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:40.832825 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.832809 4603 scope.go:122] "RemoveContainer" containerID="bf290b19a878dbf3b3491b1985b16ba5030b0285dfeaf04cdee0238993694388" Oct 30 18:25:40.834264 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.834240 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:40.834361 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.834293 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:40.834361 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.834299 4603 generic.go:356] "Generic (PLEG): container finished" podID="3e865ee8-6656-40a4-beca-34d6e43d4b8c" containerID="927bde0819ecda487e1d05e1c7ec9ecb1f29a963e8991c98d10f681171bb61e1" exitCode=0 Oct 30 18:25:40.834361 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.834323 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:40.834534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.834378 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" event={"ID":"3e865ee8-6656-40a4-beca-34d6e43d4b8c","Type":"ContainerDied","Data":"927bde0819ecda487e1d05e1c7ec9ecb1f29a963e8991c98d10f681171bb61e1"} Oct 30 18:25:40.834534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.834438 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" event={"ID":"3e865ee8-6656-40a4-beca-34d6e43d4b8c","Type":"ContainerDied","Data":"61464429c5d8e4a9143b2e3fdd59a48232b9db66b5fc90c31b6de6e9e4e557fa"} Oct 30 18:25:40.834534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.834463 4603 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61464429c5d8e4a9143b2e3fdd59a48232b9db66b5fc90c31b6de6e9e4e557fa" Oct 30 18:25:40.837771 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.837634 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" event={"ID":"933bdf35-df63-46f1-9cb5-62d484a4ecb0","Type":"ContainerStarted","Data":"7fdeb3ad4b7042af2400ed025bdf7f20e3511bd967a846da3b39100dd7bfd7bb"} Oct 30 18:25:40.840439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.840421 4603 generic.go:356] "Generic (PLEG): container finished" podID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerID="bb57dd61d380ff4d5f80723b7a4b6801e9620be7bdf496c960b11333446b6337" exitCode=0 Oct 30 18:25:40.840547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.840493 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" event={"ID":"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd","Type":"ContainerDied","Data":"bb57dd61d380ff4d5f80723b7a4b6801e9620be7bdf496c960b11333446b6337"} Oct 30 18:25:40.840979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.840958 4603 scope.go:122] "RemoveContainer" containerID="bb57dd61d380ff4d5f80723b7a4b6801e9620be7bdf496c960b11333446b6337" Oct 30 18:25:40.842650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.842611 4603 generic.go:356] "Generic (PLEG): container finished" podID="2546a5f3-a662-42f8-91a2-0870880c3291" containerID="80167c0ce897f4f7e70aab6ec21b6448441828671060fbd79488eb94d2d07f16" exitCode=0 Oct 30 18:25:40.842650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.842640 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" event={"ID":"2546a5f3-a662-42f8-91a2-0870880c3291","Type":"ContainerDied","Data":"80167c0ce897f4f7e70aab6ec21b6448441828671060fbd79488eb94d2d07f16"} Oct 30 18:25:40.843662 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.843641 4603 scope.go:122] "RemoveContainer" containerID="80167c0ce897f4f7e70aab6ec21b6448441828671060fbd79488eb94d2d07f16" Oct 30 18:25:40.843810 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.843782 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:40.843883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.843866 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:40.843939 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.843912 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:40.844959 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.844937 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" event={"ID":"c718da34-63db-4d45-84e9-bdc0e45ad26b","Type":"ContainerStarted","Data":"231bf7aa628242cab4811863f552d4e72e7212cd9af0ac964b48d8cd246821a7"} Oct 30 18:25:40.845035 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.844982 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" event={"ID":"c718da34-63db-4d45-84e9-bdc0e45ad26b","Type":"ContainerStarted","Data":"206fd4141b83910d3a36bfaa2e3039ca34ff3119673cbd875b05ee6eea70f453"} Oct 30 18:25:40.845136 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.845116 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" podUID="c718da34-63db-4d45-84e9-bdc0e45ad26b" containerName="istio-proxy" containerID="cri-o://231bf7aa628242cab4811863f552d4e72e7212cd9af0ac964b48d8cd246821a7" gracePeriod=30 Oct 30 18:25:40.846092 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.846075 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:25:40.850814 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.850681 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:40.850814 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.850742 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:40.850814 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.850774 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:40.901683 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.901656 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istiod-ca-cert\") pod \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " Oct 30 18:25:40.901991 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.901972 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-data\") pod \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " Oct 30 18:25:40.902155 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.902142 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-token\" (UniqueName: \"kubernetes.io/projected/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-token\") pod \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " Oct 30 18:25:40.902263 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.902250 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-workload-socket\") pod \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " Oct 30 18:25:40.902540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.902525 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e865ee8-6656-40a4-beca-34d6e43d4b8c-kube-api-access-tphzm\" (UniqueName: \"kubernetes.io/projected/3e865ee8-6656-40a4-beca-34d6e43d4b8c-kube-api-access-tphzm\") pod \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " Oct 30 18:25:40.902675 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.902660 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-podinfo\") pod \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " Oct 30 18:25:40.902783 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.902771 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-workload-certs\") pod \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " Oct 30 18:25:40.902957 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.902945 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-envoy\") pod \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " Oct 30 18:25:40.903068 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.903056 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-credential-socket\") pod \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\" (UID: \"3e865ee8-6656-40a4-beca-34d6e43d4b8c\") " Oct 30 18:25:40.903437 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.903418 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.903720 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.903699 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30517af-31fe-458a-8b17-40bc5ab61274-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.904025 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.903992 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.904730 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.902546 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-data" pod "3e865ee8-6656-40a4-beca-34d6e43d4b8c" (UID: "3e865ee8-6656-40a4-beca-34d6e43d4b8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:40.904730 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.903621 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istiod-ca-cert" pod "3e865ee8-6656-40a4-beca-34d6e43d4b8c" (UID: "3e865ee8-6656-40a4-beca-34d6e43d4b8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:40.904945 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.904411 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-workload-certs" pod "3e865ee8-6656-40a4-beca-34d6e43d4b8c" (UID: "3e865ee8-6656-40a4-beca-34d6e43d4b8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:40.905050 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.905023 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-workload-socket" pod "3e865ee8-6656-40a4-beca-34d6e43d4b8c" (UID: "3e865ee8-6656-40a4-beca-34d6e43d4b8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:40.905050 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.905023 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-credential-socket" pod "3e865ee8-6656-40a4-beca-34d6e43d4b8c" (UID: "3e865ee8-6656-40a4-beca-34d6e43d4b8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:40.905193 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.904037 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.906302 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.906265 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.906728 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.906710 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30517af-31fe-458a-8b17-40bc5ab61274-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.907026 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.907003 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.907159 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.907129 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.907249 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.907180 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8jdz\" (UniqueName: \"kubernetes.io/projected/a30517af-31fe-458a-8b17-40bc5ab61274-kube-api-access-w8jdz\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.907249 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.907231 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30517af-31fe-458a-8b17-40bc5ab61274-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.907538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.907475 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:40.907538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.907493 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:40.907538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.907531 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:40.908372 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.907546 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:40.908372 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.907561 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:40.909124 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.908765 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.909124 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.908933 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.909411 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.906720 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.915572 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.909739 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30517af-31fe-458a-8b17-40bc5ab61274-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.915572 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.909849 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-podinfo" pod "3e865ee8-6656-40a4-beca-34d6e43d4b8c" (UID: "3e865ee8-6656-40a4-beca-34d6e43d4b8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:40.927038 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.926798 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.927038 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.926932 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-token" pod "3e865ee8-6656-40a4-beca-34d6e43d4b8c" (UID: "3e865ee8-6656-40a4-beca-34d6e43d4b8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:40.927038 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.926989 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-envoy" pod "3e865ee8-6656-40a4-beca-34d6e43d4b8c" (UID: "3e865ee8-6656-40a4-beca-34d6e43d4b8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:40.932008 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.931977 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30517af-31fe-458a-8b17-40bc5ab61274-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.932186 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.932153 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e865ee8-6656-40a4-beca-34d6e43d4b8c-kube-api-access-tphzm" pod "3e865ee8-6656-40a4-beca-34d6e43d4b8c" (UID: "3e865ee8-6656-40a4-beca-34d6e43d4b8c"). InnerVolumeSpecName "kube-api-access-tphzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:40.932968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.932911 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30517af-31fe-458a-8b17-40bc5ab61274-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:40.944568 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.941991 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" podStartSLOduration=1.9419740760000002 podStartE2EDuration="1.941974076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:40.940420043 +0000 UTC m=+2001.655989917" watchObservedRunningTime="2026-10-30 18:25:40.941974076 +0000 UTC m=+2001.657543951" Oct 30 18:25:40.958990 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:40.958960 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8jdz\" (UniqueName: \"kubernetes.io/projected/a30517af-31fe-458a-8b17-40bc5ab61274-kube-api-access-w8jdz\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:41.009060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:41.008974 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:41.009060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:41.009006 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:41.009060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:41.009021 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tphzm\" (UniqueName: \"kubernetes.io/projected/3e865ee8-6656-40a4-beca-34d6e43d4b8c-kube-api-access-tphzm\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:41.009060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:41.009034 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e865ee8-6656-40a4-beca-34d6e43d4b8c-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:41.158282 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:41.158199 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:41.260921 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:41.260625 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:41.944528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:41.940793 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" event={"ID":"fa0c0084-b530-436d-936a-b94ebc16fe03","Type":"ContainerStarted","Data":"b367620c81e1e3de8d92df072d940902a6b9379a67a8678eca98576dc8c9af5c"} Oct 30 18:25:41.976439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:41.976289 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" event={"ID":"933bdf35-df63-46f1-9cb5-62d484a4ecb0","Type":"ContainerStarted","Data":"8dad318ed6b99a4b62a398986f843fb07dd5edc6b153a67f34c5c2c6d045fcfd"} Oct 30 18:25:42.020007 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.014940 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" event={"ID":"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd","Type":"ContainerStarted","Data":"992e118a84566c7df7c84dd5b94ec3cf48e2b89363ad1ff44bddc6c0d9dfa7d4"} Oct 30 18:25:42.113770 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.113735 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" event={"ID":"2546a5f3-a662-42f8-91a2-0870880c3291","Type":"ContainerStarted","Data":"c8b35cd2e9b0d1e141eaafe89a64fff170c539a0c384e6bdc5e770e2006dbe32"} Oct 30 18:25:42.204285 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.193626 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl" Oct 30 18:25:42.204285 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.193795 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" event={"ID":"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af","Type":"ContainerStarted","Data":"db1c2d3bdf6b84e095aebc306c15e9f4cbdf6b12f94e8a22df62f7a316156ab0"} Oct 30 18:25:42.204285 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.200682 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx"] Oct 30 18:25:42.241231 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.241206 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:25:42.247525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.245656 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:42.247525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.245765 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:42.247525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.245803 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:42.254519 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.251028 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 18:25:42.254519 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.251077 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" podUID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 18:25:42.254519 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.251262 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:25:42.324165 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.324134 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:42.325539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.325479 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:25:42.328322 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.328240 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" podUID="933bdf35-df63-46f1-9cb5-62d484a4ecb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 18:25:42.330292 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.326308 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:42.491220 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.487622 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:25:42.508691 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.508600 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:25:42.513528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.511413 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:25:42.513528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.511473 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 18:25:42.513528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.511929 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:25:42.513528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.511968 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" podUID="2546a5f3-a662-42f8-91a2-0870880c3291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 18:25:42.550014 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.549340 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" podStartSLOduration=3.549319099 podStartE2EDuration="3.549319099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:42.547775357 +0000 UTC m=+2003.263345229" watchObservedRunningTime="2026-10-30 18:25:42.549319099 +0000 UTC m=+2003.264888978" Oct 30 18:25:42.612396 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.612300 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:42.795443 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.788629 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:42.862601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.862444 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:25:42.882289 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.882256 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 18:25:42.895538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.895482 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" podUID="fa0c0084-b530-436d-936a-b94ebc16fe03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 18:25:42.918789 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.895377 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl"] Oct 30 18:25:42.963762 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:42.963722 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-l7czl"] Oct 30 18:25:43.122588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:43.097678 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:43.263547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:43.253033 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 18:25:43.263547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:43.253088 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" podUID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 18:25:43.301496 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:43.299852 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" event={"ID":"a30517af-31fe-458a-8b17-40bc5ab61274","Type":"ContainerStarted","Data":"3dd642d13619bdb151fe0bb4378962353668ac597acbf8f8af8bfe22eb012bd7"} Oct 30 18:25:43.301496 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:43.299913 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" event={"ID":"a30517af-31fe-458a-8b17-40bc5ab61274","Type":"ContainerStarted","Data":"dbd3181e1e1fcdefe84c860ae2cbcc767f3792e96dd71e26000a7ec4f52f5770"} Oct 30 18:25:43.314070 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:43.313997 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:25:43.314375 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:43.314322 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" podUID="933bdf35-df63-46f1-9cb5-62d484a4ecb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 18:25:43.373673 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:43.373553 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" podStartSLOduration=3.373528995 podStartE2EDuration="3.373528995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:43.36249056 +0000 UTC m=+2004.078060431" watchObservedRunningTime="2026-10-30 18:25:43.373528995 +0000 UTC m=+2004.089098870" Oct 30 18:25:43.836531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:43.836079 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:43.877601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:43.871385 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:25:43.877601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:43.871461 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 18:25:43.877601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:43.876845 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 18:25:43.877601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:43.876907 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" podUID="fa0c0084-b530-436d-936a-b94ebc16fe03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 18:25:43.895962 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:43.895914 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:25:44.010659 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.010595 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:25:44.037537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.017063 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:25:44.037537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.017384 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" podUID="2546a5f3-a662-42f8-91a2-0870880c3291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 18:25:44.090992 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.090893 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:44.092047 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.092016 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:25:44.174850 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.174809 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 18:25:44.175119 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.175093 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" podUID="a30517af-31fe-458a-8b17-40bc5ab61274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 18:25:44.191535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.189277 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e865ee8-6656-40a4-beca-34d6e43d4b8c" path="/var/lib/kubelet/pods/3e865ee8-6656-40a4-beca-34d6e43d4b8c/volumes" Oct 30 18:25:44.191535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.189842 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:44.252645 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.252608 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 18:25:44.252944 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.252913 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" podUID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 18:25:44.291529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.288600 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:44.307772 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.307732 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:25:44.320049 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.320014 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:25:44.320315 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.320290 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" podUID="933bdf35-df63-46f1-9cb5-62d484a4ecb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 18:25:44.474966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.474874 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:25:44.475216 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.475188 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 18:25:44.510237 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.510200 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:25:44.510558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.510528 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" podUID="2546a5f3-a662-42f8-91a2-0870880c3291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 18:25:44.837529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.836612 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:25:44.837529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.837297 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:25:44.862530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.853653 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:44.874570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.863885 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 18:25:44.874570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.863948 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" podUID="fa0c0084-b530-436d-936a-b94ebc16fe03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 18:25:44.981525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.917297 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:25:44.981525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.917914 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:25:44.981525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:44.948604 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Oct 30 18:25:45.085530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.058092 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:25:45.085530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.058730 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:25:45.085530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.064864 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:45.223594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.092586 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:45.223594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.092993 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:25:45.223594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.093577 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:25:45.223594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.093634 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:25:45.223594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.162908 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 18:25:45.223594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.162966 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" podUID="a30517af-31fe-458a-8b17-40bc5ab61274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 18:25:45.311549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.284872 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 18:25:45.311549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.284970 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" podUID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 18:25:45.336531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.325180 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:45.343532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.326732 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:45.387534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.338287 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:25:45.387534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.343768 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" podUID="933bdf35-df63-46f1-9cb5-62d484a4ecb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 18:25:45.387534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.339490 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:45.387534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.345073 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:25:45.387534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.345231 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:25:45.510734 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.510632 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:45.621629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.551699 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:25:45.621629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.551778 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" podUID="2546a5f3-a662-42f8-91a2-0870880c3291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 18:25:45.638588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.637890 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:25:45.638588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.637947 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 18:25:46.010230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.898738 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:25:46.010230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.920663 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:25:46.010230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.951668 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 18:25:46.010230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.970226 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:25:46.010230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.976019 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" podUID="fa0c0084-b530-436d-936a-b94ebc16fe03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 18:25:46.010230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:45.984602 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:25:46.069530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:46.068527 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:25:46.096532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:46.094457 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:46.133532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:46.098570 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:25:46.205623 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:46.201688 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 18:25:46.205623 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:46.201753 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" podUID="a30517af-31fe-458a-8b17-40bc5ab61274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 18:25:46.286416 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:46.270627 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 18:25:46.286416 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:46.270687 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" podUID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 18:25:46.353533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:46.346080 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:46.353533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:46.346399 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:25:46.397524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:46.397472 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:25:46.414856 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:46.414817 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" podUID="933bdf35-df63-46f1-9cb5-62d484a4ecb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 18:25:46.536387 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:46.536352 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:25:46.536763 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:46.536689 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" podUID="2546a5f3-a662-42f8-91a2-0870880c3291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 18:25:46.602700 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:46.602662 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:25:46.604301 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:46.604268 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 18:25:46.621053 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:46.621015 4603 generic.go:356] "Generic (PLEG): container finished" podID="c718da34-63db-4d45-84e9-bdc0e45ad26b" containerID="231bf7aa628242cab4811863f552d4e72e7212cd9af0ac964b48d8cd246821a7" exitCode=0 Oct 30 18:25:46.621317 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:46.621294 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" event={"ID":"c718da34-63db-4d45-84e9-bdc0e45ad26b","Type":"ContainerDied","Data":"231bf7aa628242cab4811863f552d4e72e7212cd9af0ac964b48d8cd246821a7"} Oct 30 18:25:46.865587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:46.861930 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 18:25:46.865587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:46.861996 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" podUID="fa0c0084-b530-436d-936a-b94ebc16fe03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 18:25:46.948234 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:46.948192 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:46.964588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:46.964549 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:46.992042 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:46.992001 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:47.071654 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:47.071605 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:47.138700 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:47.138603 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:25:47.195531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:47.145707 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:25:47.215016 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:47.187190 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:47.372565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:47.243343 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 18:25:47.373091 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:47.373057 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" podUID="a30517af-31fe-458a-8b17-40bc5ab61274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 18:25:47.388906 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:47.366622 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:47.389484 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:47.366672 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded" Oct 30 18:25:47.411672 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:47.366694 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:25:47.412177 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:47.268417 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:47.412466 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:47.268782 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 18:25:47.412690 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:47.412661 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" podUID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 18:25:47.412844 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:47.311902 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:25:47.412970 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:47.412947 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" podUID="933bdf35-df63-46f1-9cb5-62d484a4ecb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 18:25:47.529963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:47.529927 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:25:47.530206 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:47.530180 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" podUID="2546a5f3-a662-42f8-91a2-0870880c3291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 18:25:47.532176 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:47.532150 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:25:47.532404 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:47.532379 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 18:25:47.598117 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:47.598081 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:25:47.821447 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:47.700570 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:47.861463 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:47.861427 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 18:25:47.861763 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:47.861733 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" podUID="fa0c0084-b530-436d-936a-b94ebc16fe03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 18:25:48.012388 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.012354 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:48.106601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.068749 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:25:48.106601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.069201 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:25:48.106601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.069555 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:25:48.169534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.168252 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": context deadline exceeded" Oct 30 18:25:48.183632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.176757 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 18:25:48.183632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.176814 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" podUID="a30517af-31fe-458a-8b17-40bc5ab61274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 18:25:48.185766 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.185724 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:48.216452 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.214942 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:48.216699 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.174615 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:48.274377 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.274341 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 18:25:48.274806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.274773 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" podUID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 18:25:48.336727 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.336691 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:48.374776 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.374687 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:25:48.375072 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.375048 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" podUID="933bdf35-df63-46f1-9cb5-62d484a4ecb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 18:25:48.389297 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.389258 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:48.471539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.448685 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 30 18:25:48.471539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.464853 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:25:48.537645 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.474295 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-credential-socket\") pod \"c718da34-63db-4d45-84e9-bdc0e45ad26b\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " Oct 30 18:25:48.569540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.558733 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c718da34-63db-4d45-84e9-bdc0e45ad26b-kube-api-access-6cxhs\" (UniqueName: \"kubernetes.io/projected/c718da34-63db-4d45-84e9-bdc0e45ad26b-kube-api-access-6cxhs\") pod \"c718da34-63db-4d45-84e9-bdc0e45ad26b\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " Oct 30 18:25:48.569540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.558822 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c718da34-63db-4d45-84e9-bdc0e45ad26b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c718da34-63db-4d45-84e9-bdc0e45ad26b-istiod-ca-cert\") pod \"c718da34-63db-4d45-84e9-bdc0e45ad26b\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " Oct 30 18:25:48.569540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.558911 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-data\") pod \"c718da34-63db-4d45-84e9-bdc0e45ad26b\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " Oct 30 18:25:48.569540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.559058 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-envoy\") pod \"c718da34-63db-4d45-84e9-bdc0e45ad26b\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " Oct 30 18:25:48.569540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.559142 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-podinfo\") pod \"c718da34-63db-4d45-84e9-bdc0e45ad26b\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " Oct 30 18:25:48.569540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.559300 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-workload-socket\") pod \"c718da34-63db-4d45-84e9-bdc0e45ad26b\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " Oct 30 18:25:48.569540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.559358 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-token\" (UniqueName: \"kubernetes.io/projected/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-token\") pod \"c718da34-63db-4d45-84e9-bdc0e45ad26b\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " Oct 30 18:25:48.569540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.559419 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-workload-certs\") pod \"c718da34-63db-4d45-84e9-bdc0e45ad26b\" (UID: \"c718da34-63db-4d45-84e9-bdc0e45ad26b\") " Oct 30 18:25:48.569540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.507359 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:25:48.569540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.563710 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 18:25:48.592048 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.592010 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-workload-certs" pod "c718da34-63db-4d45-84e9-bdc0e45ad26b" (UID: "c718da34-63db-4d45-84e9-bdc0e45ad26b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:48.593428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.593166 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-data" pod "c718da34-63db-4d45-84e9-bdc0e45ad26b" (UID: "c718da34-63db-4d45-84e9-bdc0e45ad26b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:48.593793 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.593774 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c718da34-63db-4d45-84e9-bdc0e45ad26b-istiod-ca-cert" pod "c718da34-63db-4d45-84e9-bdc0e45ad26b" (UID: "c718da34-63db-4d45-84e9-bdc0e45ad26b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:48.593923 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.508381 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:25:48.594078 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.594044 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" podUID="2546a5f3-a662-42f8-91a2-0870880c3291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 18:25:48.594670 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.509640 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-credential-socket" pod "c718da34-63db-4d45-84e9-bdc0e45ad26b" (UID: "c718da34-63db-4d45-84e9-bdc0e45ad26b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:48.741746 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.741654 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Oct 30 18:25:48.773760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.751244 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c718da34-63db-4d45-84e9-bdc0e45ad26b-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:48.773760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.751275 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:48.773760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.751291 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:48.773760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.751312 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:48.834537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.796256 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-envoy" pod "c718da34-63db-4d45-84e9-bdc0e45ad26b" (UID: "c718da34-63db-4d45-84e9-bdc0e45ad26b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:48.854514 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.854463 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:48.854813 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.854786 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-workload-socket" pod "c718da34-63db-4d45-84e9-bdc0e45ad26b" (UID: "c718da34-63db-4d45-84e9-bdc0e45ad26b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:48.951950 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.951910 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 18:25:48.958539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.952227 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" podUID="fa0c0084-b530-436d-936a-b94ebc16fe03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 18:25:48.974863 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:48.961847 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c718da34-63db-4d45-84e9-bdc0e45ad26b-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.123652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.078062 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:25:49.123652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.078472 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:25:49.123652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.078828 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:25:49.206700 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.157271 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c718da34-63db-4d45-84e9-bdc0e45ad26b-kube-api-access-6cxhs" pod "c718da34-63db-4d45-84e9-bdc0e45ad26b" (UID: "c718da34-63db-4d45-84e9-bdc0e45ad26b"). InnerVolumeSpecName "kube-api-access-6cxhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:49.238528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.202651 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:49.238717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.238616 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:49.239833 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.173362 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-podinfo" pod "c718da34-63db-4d45-84e9-bdc0e45ad26b" (UID: "c718da34-63db-4d45-84e9-bdc0e45ad26b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:49.240011 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.182824 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 18:25:49.243547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.240158 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" podUID="a30517af-31fe-458a-8b17-40bc5ab61274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 18:25:49.262266 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.247729 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:49.262788 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.250318 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 18:25:49.263028 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.262989 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" podUID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 18:25:49.308571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.289713 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-token" pod "c718da34-63db-4d45-84e9-bdc0e45ad26b" (UID: "c718da34-63db-4d45-84e9-bdc0e45ad26b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:49.308571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.233622 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cxhs\" (UniqueName: \"kubernetes.io/projected/c718da34-63db-4d45-84e9-bdc0e45ad26b-kube-api-access-6cxhs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.308571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.289892 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" Oct 30 18:25:49.312128 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.311991 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:25:49.312128 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.312050 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" podUID="933bdf35-df63-46f1-9cb5-62d484a4ecb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 18:25:49.316869 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.312386 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7" event={"ID":"c718da34-63db-4d45-84e9-bdc0e45ad26b","Type":"ContainerDied","Data":"206fd4141b83910d3a36bfaa2e3039ca34ff3119673cbd875b05ee6eea70f453"} Oct 30 18:25:49.316869 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.312538 4603 scope.go:122] "RemoveContainer" containerID="231bf7aa628242cab4811863f552d4e72e7212cd9af0ac964b48d8cd246821a7" Oct 30 18:25:49.401610 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.390179 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:49.401610 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.392257 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.401610 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.392282 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c718da34-63db-4d45-84e9-bdc0e45ad26b-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.479214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.478982 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:49.519209 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.518599 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:25:49.519209 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.518655 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 18:25:49.519209 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.519065 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:25:49.519209 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.519102 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" podUID="2546a5f3-a662-42f8-91a2-0870880c3291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 18:25:49.906529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.871261 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:25:49.914230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.911034 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 18:25:49.914460 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.914435 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" podUID="fa0c0084-b530-436d-936a-b94ebc16fe03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 18:25:50.017540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:49.980762 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7"] Oct 30 18:25:50.071531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.032724 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-tdkh7"] Oct 30 18:25:50.071938 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.066167 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:50.079679 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.079624 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:50.080688 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.080080 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:50.081089 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.080215 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:50.172668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.172571 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 18:25:50.172941 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.172910 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" podUID="a30517af-31fe-458a-8b17-40bc5ab61274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 18:25:50.240536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.239745 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:50.241342 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.241312 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:50.270256 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.270200 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:50.284529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.277654 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 18:25:50.284529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.277721 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" podUID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 18:25:50.285842 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.285810 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:50.311958 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.311923 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:25:50.312246 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.312211 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" podUID="933bdf35-df63-46f1-9cb5-62d484a4ecb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 18:25:50.316715 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.316670 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:50.393631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.393066 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:50.521654 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.491189 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t"] Oct 30 18:25:50.550528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.531657 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" podUID="fa0c0084-b530-436d-936a-b94ebc16fe03" containerName="istio-proxy" containerID="cri-o://b367620c81e1e3de8d92df072d940902a6b9379a67a8678eca98576dc8c9af5c" gracePeriod=30 Oct 30 18:25:50.550528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.543977 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:25:50.550528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.544027 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" podUID="2546a5f3-a662-42f8-91a2-0870880c3291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 18:25:50.570527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.570473 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:50.577676 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.577636 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:25:50.578735 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.577875 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 18:25:50.580630 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:50.580494 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:51.086994 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:51.086960 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:25:51.089311 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:51.088032 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:51.115586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:51.088069 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:25:51.116664 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:51.088091 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:25:51.168545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:51.168486 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 18:25:51.168806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:51.168783 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" podUID="a30517af-31fe-458a-8b17-40bc5ab61274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 18:25:51.259529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:51.243517 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:51.259529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:51.244225 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:25:51.335547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:51.330598 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:51.349734 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:51.349648 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:25:51.349991 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:51.349962 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" podUID="933bdf35-df63-46f1-9cb5-62d484a4ecb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 18:25:51.397185 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:51.397143 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:51.449726 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:51.449686 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 18:25:51.450023 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:51.449994 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" podUID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 18:25:51.479023 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:51.478965 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:25:51.483611 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:51.483574 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 18:25:51.569718 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:51.509603 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:25:51.570117 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:51.570086 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" podUID="2546a5f3-a662-42f8-91a2-0870880c3291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 18:25:51.723580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:51.707100 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c718da34-63db-4d45-84e9-bdc0e45ad26b" path="/var/lib/kubelet/pods/c718da34-63db-4d45-84e9-bdc0e45ad26b/volumes" Oct 30 18:25:52.102539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:52.068725 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:52.113203 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:52.113123 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:52.120955 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:52.120891 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:52.122728 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:52.122680 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:25:52.179538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:52.173568 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 18:25:52.179538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:52.173628 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" podUID="a30517af-31fe-458a-8b17-40bc5ab61274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 18:25:52.213548 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:25:52.213495 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:25:52.254040 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:52.253997 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 18:25:52.254288 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:52.254260 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" podUID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 18:25:52.254871 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:52.254840 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:52.255316 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:52.255296 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:25:52.315011 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:52.314967 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:25:52.315627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:52.315595 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" podUID="933bdf35-df63-46f1-9cb5-62d484a4ecb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 18:25:52.331006 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:52.330970 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:52.400764 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:52.400649 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:52.423851 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:52.423794 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:52.474000 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:52.473967 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:25:52.474318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:52.474274 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 18:25:52.514052 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:52.514009 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:25:52.514361 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:52.514320 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" podUID="2546a5f3-a662-42f8-91a2-0870880c3291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 18:25:53.126808 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:53.126760 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:25:53.127444 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:53.127413 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:25:53.171949 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:53.171902 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:53.173544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:53.173490 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 18:25:53.197841 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:53.197796 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" podUID="a30517af-31fe-458a-8b17-40bc5ab61274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 18:25:53.249526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:53.249049 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 18:25:53.249526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:53.249122 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" podUID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 18:25:53.272174 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:53.272130 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:53.274063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:53.274022 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:53.332078 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:53.322573 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:25:53.408862 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:53.394642 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" podUID="933bdf35-df63-46f1-9cb5-62d484a4ecb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 18:25:53.408862 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:53.331718 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:53.431530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:53.419266 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:25:53.467526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:53.440635 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:53.514529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:53.512464 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:25:53.514529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:53.512547 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 18:25:53.531932 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:53.531894 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:25:53.532228 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:53.532198 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" podUID="2546a5f3-a662-42f8-91a2-0870880c3291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 18:25:53.538736 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:53.538376 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:25:53.564859 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:53.564807 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:25:53.823534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:53.823291 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:54.129587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:54.128663 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:25:54.129587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:54.129121 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:54.178530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:54.163731 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 18:25:54.178530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:54.163808 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" podUID="a30517af-31fe-458a-8b17-40bc5ab61274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 18:25:54.179649 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:54.179609 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:54.258419 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:54.258383 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 18:25:54.258740 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:54.258709 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" podUID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 18:25:54.291542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:54.290607 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:54.318965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:54.318923 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:25:54.320519 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:54.320465 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" podUID="933bdf35-df63-46f1-9cb5-62d484a4ecb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 18:25:54.335411 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:54.334078 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:54.400720 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:54.399624 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:54.419544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:54.419484 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:54.444636 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:54.444591 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:54.474970 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:54.474935 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:25:54.475253 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:54.475211 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 18:25:54.521891 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:54.521857 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:25:54.522164 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:54.522128 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" podUID="2546a5f3-a662-42f8-91a2-0870880c3291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 18:25:54.623979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:54.623720 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:54.998529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:54.978321 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq"] Oct 30 18:25:55.026732 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.026702 4603 cpu_manager.go:414] "containerMap: removing container" podUID="3e865ee8-6656-40a4-beca-34d6e43d4b8c" containerName="istio-proxy" Oct 30 18:25:55.027032 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.027013 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e865ee8-6656-40a4-beca-34d6e43d4b8c" containerName="istio-proxy" Oct 30 18:25:55.027294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.027277 4603 cpu_manager.go:414] "containerMap: removing container" podUID="c718da34-63db-4d45-84e9-bdc0e45ad26b" containerName="istio-proxy" Oct 30 18:25:55.030568 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.030542 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c718da34-63db-4d45-84e9-bdc0e45ad26b" containerName="istio-proxy" Oct 30 18:25:55.034616 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.034597 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e865ee8-6656-40a4-beca-34d6e43d4b8c" containerName="istio-proxy" Oct 30 18:25:55.034750 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.034738 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="c718da34-63db-4d45-84e9-bdc0e45ad26b" containerName="istio-proxy" Oct 30 18:25:55.137220 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.136106 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:55.137220 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.136462 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:55.139818 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.139771 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq"] Oct 30 18:25:55.140150 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.140072 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.180447 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.179942 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:55.185078 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.184642 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 18:25:55.185078 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.184696 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" podUID="a30517af-31fe-458a-8b17-40bc5ab61274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 18:25:55.282481 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.282435 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-credential-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.282877 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.282855 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-workload-certs\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.283060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.283045 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjkhw\" (UniqueName: \"kubernetes.io/projected/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-kube-api-access-rjkhw\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.283206 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.283191 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-envoy\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.283334 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.283321 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-podinfo\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.283607 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.283590 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-token\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.283754 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.283739 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-data\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.283891 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.283877 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-workload-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.284086 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.284070 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istiod-ca-cert\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.295820 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.295774 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:25:55.311276 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.311240 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:55.334339 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.334300 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:55.396629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.385878 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-credential-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.396629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.386027 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-workload-certs\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.396629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.386068 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjkhw\" (UniqueName: \"kubernetes.io/projected/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-kube-api-access-rjkhw\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.396629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.386130 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-envoy\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.396629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.386156 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-podinfo\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.396629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.386280 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-token\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.396629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.386313 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-data\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.396629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.386358 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-workload-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.396629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.386467 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istiod-ca-cert\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.396629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.387257 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istiod-ca-cert\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.396629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.387921 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-credential-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.396629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.388324 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-workload-certs\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.405412 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.405378 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-data\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.422685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.420139 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:25:55.423221 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.421427 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:25:55.468811 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.421701 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-workload-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.469584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.448284 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:55.493431 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.473121 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:55.505544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.501033 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Oct 30 18:25:55.593490 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.530191 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-podinfo\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.593490 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.556011 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-envoy\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.689417 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.689387 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:25:55.689690 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.689677 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:25:55.694722 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.694698 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjkhw\" (UniqueName: \"kubernetes.io/projected/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-kube-api-access-rjkhw\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.695418 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.695401 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-token\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:55.696401 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.695962 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:25:55.716651 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.713489 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:25:55.844597 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.842777 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:55.870532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.870263 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:25:55.918366 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.918242 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:25:56.007599 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.952766 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:25:56.007599 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:55.986636 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:56.061541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:56.038692 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:25:56.073191 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:56.071674 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:25:56.073191 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:56.072347 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:25:56.117193 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:56.108935 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:25:56.141483 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:56.140640 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:56.141483 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:56.140971 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:25:56.238924 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:56.160717 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:56.238924 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:56.163622 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 18:25:56.238924 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:56.163667 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" podUID="a30517af-31fe-458a-8b17-40bc5ab61274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 18:25:56.238924 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:56.169664 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:25:56.238924 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:56.225758 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:25:56.238924 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:56.226759 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:25:56.356968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:56.355965 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:25:56.531134 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:56.531104 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq"] Oct 30 18:25:56.536299 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:25:56.536268 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b16b70e_9b8b_41ce_b39c_4e3e5654f358.slice/crio-8c06981f9a5bee9e77dcb2ecd7f8c9fa96eb0dd39a166ed6bb3027df8dcb709f WatchSource:0}: Error finding container 8c06981f9a5bee9e77dcb2ecd7f8c9fa96eb0dd39a166ed6bb3027df8dcb709f: Status 404 returned error can't find the container with id 8c06981f9a5bee9e77dcb2ecd7f8c9fa96eb0dd39a166ed6bb3027df8dcb709f Oct 30 18:25:56.540001 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:56.539971 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:56.540116 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:56.540054 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:56.540116 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:56.540098 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:56.914802 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:56.914698 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" event={"ID":"6b16b70e-9b8b-41ce-b39c-4e3e5654f358","Type":"ContainerStarted","Data":"b7ad0c3456f2dd1238910d1e5c9fe3ff82c16189f38f0e37cfc28e3bcea8bb9f"} Oct 30 18:25:56.914802 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:56.914769 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" event={"ID":"6b16b70e-9b8b-41ce-b39c-4e3e5654f358","Type":"ContainerStarted","Data":"8c06981f9a5bee9e77dcb2ecd7f8c9fa96eb0dd39a166ed6bb3027df8dcb709f"} Oct 30 18:25:56.962586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:56.962539 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" podStartSLOduration=2.962525776 podStartE2EDuration="2.962525776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:56.958576258 +0000 UTC m=+2017.674146150" watchObservedRunningTime="2026-10-30 18:25:56.962525776 +0000 UTC m=+2017.678095647" Oct 30 18:25:56.988631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:56.988601 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:25:57.154211 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:57.154164 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:57.214924 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:57.214838 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:57.244527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:57.238652 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:25:57.523962 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:57.523918 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:58.014588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:58.013627 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 18:25:58.014588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:58.013691 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" podUID="6b16b70e-9b8b-41ce-b39c-4e3e5654f358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 18:25:58.929923 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:58.929887 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:58.993191 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:58.993150 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 18:25:58.993482 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:58.993449 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" podUID="6b16b70e-9b8b-41ce-b39c-4e3e5654f358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 18:25:59.427870 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:59.427835 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:59.583170 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:25:59.578868 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:00.112528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:00.030085 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 18:26:00.112528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:00.030190 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" podUID="6b16b70e-9b8b-41ce-b39c-4e3e5654f358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 18:26:00.952418 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:00.952368 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:00.988426 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:00.988398 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 18:26:00.988561 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:00.988457 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" podUID="6b16b70e-9b8b-41ce-b39c-4e3e5654f358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 18:26:01.276187 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.272872 4603 generic.go:356] "Generic (PLEG): container finished" podID="fa0c0084-b530-436d-936a-b94ebc16fe03" containerID="b367620c81e1e3de8d92df072d940902a6b9379a67a8678eca98576dc8c9af5c" exitCode=0 Oct 30 18:26:01.276187 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.273080 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" event={"ID":"fa0c0084-b530-436d-936a-b94ebc16fe03","Type":"ContainerDied","Data":"b367620c81e1e3de8d92df072d940902a6b9379a67a8678eca98576dc8c9af5c"} Oct 30 18:26:01.276187 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.273176 4603 scope.go:122] "RemoveContainer" containerID="bf290b19a878dbf3b3491b1985b16ba5030b0285dfeaf04cdee0238993694388" Oct 30 18:26:01.446656 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.446626 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:26:01.460288 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.460261 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-workload-socket\") pod \"fa0c0084-b530-436d-936a-b94ebc16fe03\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " Oct 30 18:26:01.460431 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.460361 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-workload-certs\") pod \"fa0c0084-b530-436d-936a-b94ebc16fe03\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " Oct 30 18:26:01.460496 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.460478 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-workload-socket" pod "fa0c0084-b530-436d-936a-b94ebc16fe03" (UID: "fa0c0084-b530-436d-936a-b94ebc16fe03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:01.460496 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.460477 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa0c0084-b530-436d-936a-b94ebc16fe03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa0c0084-b530-436d-936a-b94ebc16fe03-istiod-ca-cert\") pod \"fa0c0084-b530-436d-936a-b94ebc16fe03\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " Oct 30 18:26:01.460658 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.460563 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-credential-socket\") pod \"fa0c0084-b530-436d-936a-b94ebc16fe03\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " Oct 30 18:26:01.460715 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.460650 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-workload-certs" pod "fa0c0084-b530-436d-936a-b94ebc16fe03" (UID: "fa0c0084-b530-436d-936a-b94ebc16fe03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:01.460715 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.460682 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-token\" (UniqueName: \"kubernetes.io/projected/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-token\") pod \"fa0c0084-b530-436d-936a-b94ebc16fe03\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " Oct 30 18:26:01.460831 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.460717 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-envoy\") pod \"fa0c0084-b530-436d-936a-b94ebc16fe03\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " Oct 30 18:26:01.460831 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.460739 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa0c0084-b530-436d-936a-b94ebc16fe03-kube-api-access-rwqhx\" (UniqueName: \"kubernetes.io/projected/fa0c0084-b530-436d-936a-b94ebc16fe03-kube-api-access-rwqhx\") pod \"fa0c0084-b530-436d-936a-b94ebc16fe03\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " Oct 30 18:26:01.460831 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.460773 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-credential-socket" pod "fa0c0084-b530-436d-936a-b94ebc16fe03" (UID: "fa0c0084-b530-436d-936a-b94ebc16fe03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:01.460977 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.460825 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0c0084-b530-436d-936a-b94ebc16fe03-istiod-ca-cert" pod "fa0c0084-b530-436d-936a-b94ebc16fe03" (UID: "fa0c0084-b530-436d-936a-b94ebc16fe03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:01.460977 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.460836 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-data\") pod \"fa0c0084-b530-436d-936a-b94ebc16fe03\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " Oct 30 18:26:01.460977 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.460969 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-podinfo\") pod \"fa0c0084-b530-436d-936a-b94ebc16fe03\" (UID: \"fa0c0084-b530-436d-936a-b94ebc16fe03\") " Oct 30 18:26:01.461209 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.461184 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-data" pod "fa0c0084-b530-436d-936a-b94ebc16fe03" (UID: "fa0c0084-b530-436d-936a-b94ebc16fe03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:01.461807 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.461782 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa0c0084-b530-436d-936a-b94ebc16fe03-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.461911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.461812 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.461911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.461828 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.461911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.461843 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.461911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.461858 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.463172 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.463143 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-token" pod "fa0c0084-b530-436d-936a-b94ebc16fe03" (UID: "fa0c0084-b530-436d-936a-b94ebc16fe03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:01.463357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.463333 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-envoy" pod "fa0c0084-b530-436d-936a-b94ebc16fe03" (UID: "fa0c0084-b530-436d-936a-b94ebc16fe03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:01.463465 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.463448 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0c0084-b530-436d-936a-b94ebc16fe03-kube-api-access-rwqhx" pod "fa0c0084-b530-436d-936a-b94ebc16fe03" (UID: "fa0c0084-b530-436d-936a-b94ebc16fe03"). InnerVolumeSpecName "kube-api-access-rwqhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:01.464341 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.464326 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-podinfo" pod "fa0c0084-b530-436d-936a-b94ebc16fe03" (UID: "fa0c0084-b530-436d-936a-b94ebc16fe03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:01.563441 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.563367 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.563441 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.563393 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.563441 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.563403 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwqhx\" (UniqueName: \"kubernetes.io/projected/fa0c0084-b530-436d-936a-b94ebc16fe03-kube-api-access-rwqhx\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.563441 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.563414 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa0c0084-b530-436d-936a-b94ebc16fe03-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.987911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.987834 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 18:26:01.987911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:01.987894 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" podUID="6b16b70e-9b8b-41ce-b39c-4e3e5654f358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 18:26:02.280745 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:02.280718 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" Oct 30 18:26:02.280907 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:02.280745 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t" event={"ID":"fa0c0084-b530-436d-936a-b94ebc16fe03","Type":"ContainerDied","Data":"528d53422376f33033805b667637dfa60ba7560eb55442cd355903fb6b30a8ac"} Oct 30 18:26:02.280907 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:02.280810 4603 scope.go:122] "RemoveContainer" containerID="b367620c81e1e3de8d92df072d940902a6b9379a67a8678eca98576dc8c9af5c" Oct 30 18:26:02.353114 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:02.353085 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t"] Oct 30 18:26:02.364573 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:02.364548 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kl72t"] Oct 30 18:26:02.767611 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:02.767530 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:02.999527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:02.994976 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 18:26:02.999527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:02.995045 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" podUID="6b16b70e-9b8b-41ce-b39c-4e3e5654f358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 18:26:03.427786 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:03.427751 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa0c0084-b530-436d-936a-b94ebc16fe03" path="/var/lib/kubelet/pods/fa0c0084-b530-436d-936a-b94ebc16fe03/volumes" Oct 30 18:26:04.080950 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:04.080922 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:26:04.081992 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:04.081958 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:26:04.083447 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:04.083425 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:26:04.694719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:04.694677 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:05.992279 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:05.992246 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:06.367531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.365634 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:06.464601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.464560 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw"] Oct 30 18:26:06.465653 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.465626 4603 cpu_manager.go:414] "containerMap: removing container" podUID="fa0c0084-b530-436d-936a-b94ebc16fe03" containerName="istio-proxy" Oct 30 18:26:06.465653 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.465654 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa0c0084-b530-436d-936a-b94ebc16fe03" containerName="istio-proxy" Oct 30 18:26:06.465862 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.465710 4603 cpu_manager.go:414] "containerMap: removing container" podUID="fa0c0084-b530-436d-936a-b94ebc16fe03" containerName="istio-proxy" Oct 30 18:26:06.465862 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.465719 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa0c0084-b530-436d-936a-b94ebc16fe03" containerName="istio-proxy" Oct 30 18:26:06.465972 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.465867 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa0c0084-b530-436d-936a-b94ebc16fe03" containerName="istio-proxy" Oct 30 18:26:06.465972 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.465899 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa0c0084-b530-436d-936a-b94ebc16fe03" containerName="istio-proxy" Oct 30 18:26:06.526103 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.519158 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw" Oct 30 18:26:06.532408 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.532378 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw"] Oct 30 18:26:06.537005 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.536929 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-77db82-dockercfg-zrgw7\"" Oct 30 18:26:06.537379 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.537308 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-77db82\"" Oct 30 18:26:06.580900 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.575284 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6"] Oct 30 18:26:06.712851 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.692720 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6"] Oct 30 18:26:06.712851 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.692872 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6" Oct 30 18:26:06.712851 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.696536 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803ab509-6175-4535-a6b5-29bf877e9d72-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw\" (UID: \"803ab509-6175-4535-a6b5-29bf877e9d72\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw" Oct 30 18:26:06.712851 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.696646 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jv6s\" (UniqueName: \"kubernetes.io/projected/803ab509-6175-4535-a6b5-29bf877e9d72-kube-api-access-8jv6s\") pod \"payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw\" (UID: \"803ab509-6175-4535-a6b5-29bf877e9d72\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw" Oct 30 18:26:06.805569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.800933 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803ab509-6175-4535-a6b5-29bf877e9d72-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw\" (UID: \"803ab509-6175-4535-a6b5-29bf877e9d72\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw" Oct 30 18:26:06.805569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.801037 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jv6s\" (UniqueName: \"kubernetes.io/projected/803ab509-6175-4535-a6b5-29bf877e9d72-kube-api-access-8jv6s\") pod \"payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw\" (UID: \"803ab509-6175-4535-a6b5-29bf877e9d72\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw" Oct 30 18:26:06.822529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.815324 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803ab509-6175-4535-a6b5-29bf877e9d72-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw\" (UID: \"803ab509-6175-4535-a6b5-29bf877e9d72\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw" Oct 30 18:26:06.842530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.842362 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jv6s\" (UniqueName: \"kubernetes.io/projected/803ab509-6175-4535-a6b5-29bf877e9d72-kube-api-access-8jv6s\") pod \"payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw\" (UID: \"803ab509-6175-4535-a6b5-29bf877e9d72\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw" Oct 30 18:26:06.902163 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.902131 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5e20ee-f725-4245-be00-4df9759c4a50-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6\" (UID: \"2f5e20ee-f725-4245-be00-4df9759c4a50\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6" Oct 30 18:26:06.902356 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.902193 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rb8f\" (UniqueName: \"kubernetes.io/projected/2f5e20ee-f725-4245-be00-4df9759c4a50-kube-api-access-9rb8f\") pod \"payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6\" (UID: \"2f5e20ee-f725-4245-be00-4df9759c4a50\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6" Oct 30 18:26:06.993918 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:06.993846 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw" Oct 30 18:26:07.003403 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:07.003378 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5e20ee-f725-4245-be00-4df9759c4a50-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6\" (UID: \"2f5e20ee-f725-4245-be00-4df9759c4a50\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6" Oct 30 18:26:07.003584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:07.003424 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rb8f\" (UniqueName: \"kubernetes.io/projected/2f5e20ee-f725-4245-be00-4df9759c4a50-kube-api-access-9rb8f\") pod \"payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6\" (UID: \"2f5e20ee-f725-4245-be00-4df9759c4a50\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6" Oct 30 18:26:07.004221 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:07.004202 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5e20ee-f725-4245-be00-4df9759c4a50-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6\" (UID: \"2f5e20ee-f725-4245-be00-4df9759c4a50\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6" Oct 30 18:26:07.026733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:07.026707 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rb8f\" (UniqueName: \"kubernetes.io/projected/2f5e20ee-f725-4245-be00-4df9759c4a50-kube-api-access-9rb8f\") pod \"payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6\" (UID: \"2f5e20ee-f725-4245-be00-4df9759c4a50\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6" Oct 30 18:26:07.159236 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:07.159206 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:07.320728 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:07.320698 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6" Oct 30 18:26:07.337265 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:07.337235 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw"] Oct 30 18:26:07.338451 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:26:07.338427 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod803ab509_6175_4535_a6b5_29bf877e9d72.slice/crio-4ce4137671f96c56b9a4f5639da4b2f5a8c1b925230e865c432c0e157c00536c WatchSource:0}: Error finding container 4ce4137671f96c56b9a4f5639da4b2f5a8c1b925230e865c432c0e157c00536c: Status 404 returned error can't find the container with id 4ce4137671f96c56b9a4f5639da4b2f5a8c1b925230e865c432c0e157c00536c Oct 30 18:26:07.439211 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:07.439172 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw" event={"ID":"803ab509-6175-4535-a6b5-29bf877e9d72","Type":"ContainerStarted","Data":"48c8f9a8d6282d8d3da637130d40df86968d2eacddf7738656db68f36da975a2"} Oct 30 18:26:07.439360 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:07.439247 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw" event={"ID":"803ab509-6175-4535-a6b5-29bf877e9d72","Type":"ContainerStarted","Data":"4ce4137671f96c56b9a4f5639da4b2f5a8c1b925230e865c432c0e157c00536c"} Oct 30 18:26:07.439437 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:07.439402 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw" Oct 30 18:26:07.490987 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:07.490932 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw" podStartSLOduration=1.4909178029999999 podStartE2EDuration="1.490917803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:07.486403288 +0000 UTC m=+2028.201973164" watchObservedRunningTime="2026-10-30 18:26:07.490917803 +0000 UTC m=+2028.206487676" Oct 30 18:26:07.718592 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:07.713198 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6"] Oct 30 18:26:08.447993 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:08.447960 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6" event={"ID":"2f5e20ee-f725-4245-be00-4df9759c4a50","Type":"ContainerStarted","Data":"ae286b566c5394f52c893b33cd8af0c119fed0974fc0cabab24a626e2962494b"} Oct 30 18:26:08.448432 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:08.448014 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6" event={"ID":"2f5e20ee-f725-4245-be00-4df9759c4a50","Type":"ContainerStarted","Data":"2993b9601ef5049a1fc2abc95587b6b5c98d12dac852a8d2a8165db99125e12b"} Oct 30 18:26:08.448432 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:08.448064 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6" Oct 30 18:26:08.493181 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:08.493127 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6" podStartSLOduration=2.49311016 podStartE2EDuration="2.49311016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:08.487837859 +0000 UTC m=+2029.203407730" watchObservedRunningTime="2026-10-30 18:26:08.49311016 +0000 UTC m=+2029.208680038" Oct 30 18:26:09.036537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:09.036491 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:10.308602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:10.308565 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:10.460987 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:10.460916 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:10.746366 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:10.746283 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:12.263374 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:12.263346 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:12.896083 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:12.892965 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:13.027056 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:13.027027 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:13.180863 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:13.175085 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:13.502222 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:13.502142 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:14.856984 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:14.856938 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:16.235452 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:16.235411 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p"] Oct 30 18:26:16.235938 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:16.235647 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" podUID="6109f83b-e952-42b8-bd57-c7f53f8adadb" containerName="maas-api" containerID="cri-o://791c4bf544f35548f9338a77813978a9db37957c2f2855b024aaf39fffd7b650" gracePeriod=30 Oct 30 18:26:16.332053 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:16.332006 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:16.751900 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:16.747889 4603 generic.go:356] "Generic (PLEG): container finished" podID="6109f83b-e952-42b8-bd57-c7f53f8adadb" containerID="791c4bf544f35548f9338a77813978a9db37957c2f2855b024aaf39fffd7b650" exitCode=0 Oct 30 18:26:16.751900 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:16.747973 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" event={"ID":"6109f83b-e952-42b8-bd57-c7f53f8adadb","Type":"ContainerDied","Data":"791c4bf544f35548f9338a77813978a9db37957c2f2855b024aaf39fffd7b650"} Oct 30 18:26:16.811530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:16.802884 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" Oct 30 18:26:16.927527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:16.921016 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:16.943937 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:16.937080 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6109f83b-e952-42b8-bd57-c7f53f8adadb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6109f83b-e952-42b8-bd57-c7f53f8adadb-metrics-certs\") pod \"6109f83b-e952-42b8-bd57-c7f53f8adadb\" (UID: \"6109f83b-e952-42b8-bd57-c7f53f8adadb\") " Oct 30 18:26:16.943937 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:16.937209 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6109f83b-e952-42b8-bd57-c7f53f8adadb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6109f83b-e952-42b8-bd57-c7f53f8adadb-maas-api-tls\") pod \"6109f83b-e952-42b8-bd57-c7f53f8adadb\" (UID: \"6109f83b-e952-42b8-bd57-c7f53f8adadb\") " Oct 30 18:26:16.943937 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:16.937267 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6109f83b-e952-42b8-bd57-c7f53f8adadb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6109f83b-e952-42b8-bd57-c7f53f8adadb-openshift-service-ca\") pod \"6109f83b-e952-42b8-bd57-c7f53f8adadb\" (UID: \"6109f83b-e952-42b8-bd57-c7f53f8adadb\") " Oct 30 18:26:16.943937 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:16.937318 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6109f83b-e952-42b8-bd57-c7f53f8adadb-kube-api-access-zc2q9\" (UniqueName: \"kubernetes.io/projected/6109f83b-e952-42b8-bd57-c7f53f8adadb-kube-api-access-zc2q9\") pod \"6109f83b-e952-42b8-bd57-c7f53f8adadb\" (UID: \"6109f83b-e952-42b8-bd57-c7f53f8adadb\") " Oct 30 18:26:16.943937 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:16.939483 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6109f83b-e952-42b8-bd57-c7f53f8adadb-openshift-service-ca" pod "6109f83b-e952-42b8-bd57-c7f53f8adadb" (UID: "6109f83b-e952-42b8-bd57-c7f53f8adadb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:16.992649 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:16.980945 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6109f83b-e952-42b8-bd57-c7f53f8adadb-kube-api-access-zc2q9" pod "6109f83b-e952-42b8-bd57-c7f53f8adadb" (UID: "6109f83b-e952-42b8-bd57-c7f53f8adadb"). InnerVolumeSpecName "kube-api-access-zc2q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:16.998795 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:16.995241 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6109f83b-e952-42b8-bd57-c7f53f8adadb-metrics-certs" pod "6109f83b-e952-42b8-bd57-c7f53f8adadb" (UID: "6109f83b-e952-42b8-bd57-c7f53f8adadb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:17.030699 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:17.028319 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6109f83b-e952-42b8-bd57-c7f53f8adadb-maas-api-tls" pod "6109f83b-e952-42b8-bd57-c7f53f8adadb" (UID: "6109f83b-e952-42b8-bd57-c7f53f8adadb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:17.059949 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:17.059843 4603 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6109f83b-e952-42b8-bd57-c7f53f8adadb-openshift-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:17.059949 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:17.059884 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc2q9\" (UniqueName: \"kubernetes.io/projected/6109f83b-e952-42b8-bd57-c7f53f8adadb-kube-api-access-zc2q9\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:17.059949 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:17.059904 4603 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6109f83b-e952-42b8-bd57-c7f53f8adadb-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:17.059949 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:17.059919 4603 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6109f83b-e952-42b8-bd57-c7f53f8adadb-maas-api-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:17.068596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:17.068240 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:17.219539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:17.219206 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:17.633551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:17.632137 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:17.797318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:17.797280 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" Oct 30 18:26:17.797318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:17.797296 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p" event={"ID":"6109f83b-e952-42b8-bd57-c7f53f8adadb","Type":"ContainerDied","Data":"38e4fc672cbd62cfc2b33823389aff4b23211e69ac01ca5cad988715d11a0fe9"} Oct 30 18:26:17.797651 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:17.797404 4603 scope.go:122] "RemoveContainer" containerID="791c4bf544f35548f9338a77813978a9db37957c2f2855b024aaf39fffd7b650" Oct 30 18:26:17.849043 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:17.849012 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p"] Oct 30 18:26:17.860431 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:17.860407 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca5f33-7549dc46f8-gr72p"] Oct 30 18:26:18.450474 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:18.450449 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw" Oct 30 18:26:19.097885 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:19.097852 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:19.411042 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:19.410960 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6109f83b-e952-42b8-bd57-c7f53f8adadb" path="/var/lib/kubelet/pods/6109f83b-e952-42b8-bd57-c7f53f8adadb/volumes" Oct 30 18:26:19.468466 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:19.468444 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6" Oct 30 18:26:20.158909 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:20.158872 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:20.304812 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:20.304778 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:20.466199 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:20.466119 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx"] Oct 30 18:26:20.466388 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:20.466355 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" podUID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerName="istio-proxy" containerID="cri-o://992e118a84566c7df7c84dd5b94ec3cf48e2b89363ad1ff44bddc6c0d9dfa7d4" gracePeriod=30 Oct 30 18:26:20.539282 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:20.539244 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:20.765762 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:20.763702 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:21.931624 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:26:21.931592 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:26:22.468750 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:22.468717 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:23.761134 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:23.761101 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:23.919013 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:23.918954 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:24.192555 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:24.192522 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs"] Oct 30 18:26:24.192803 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:24.192781 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" podUID="9d3f60a0-c196-48e7-9765-b1f3d8965367" containerName="istio-proxy" containerID="cri-o://8c43fa786feb1975bf4a240378fbd2b573782e07262b74fabca7f0a43454d83c" gracePeriod=30 Oct 30 18:26:24.244866 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:24.244837 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:24.416085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:24.416053 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:24.921082 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:24.921049 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:25.748874 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.748807 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:26:25.822823 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.822793 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-kube-api-access-dhfdq\" (UniqueName: \"kubernetes.io/projected/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-kube-api-access-dhfdq\") pod \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " Oct 30 18:26:25.822991 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.822894 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istiod-ca-cert\") pod \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " Oct 30 18:26:25.823069 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.823052 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-data\") pod \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " Oct 30 18:26:25.823122 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.823107 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-token\" (UniqueName: \"kubernetes.io/projected/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-token\") pod \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " Oct 30 18:26:25.823184 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.823162 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-workload-socket\") pod \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " Oct 30 18:26:25.823240 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.823216 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-workload-certs\") pod \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " Oct 30 18:26:25.823295 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.823275 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-podinfo\") pod \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " Oct 30 18:26:25.823366 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.823351 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-credential-socket\") pod \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " Oct 30 18:26:25.823434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.823398 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-envoy\") pod \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\" (UID: \"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd\") " Oct 30 18:26:25.823493 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.823436 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-data" pod "7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" (UID: "7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:25.823594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.823571 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istiod-ca-cert" pod "7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" (UID: "7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:25.823683 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.823663 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-workload-certs" pod "7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" (UID: "7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:25.824112 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.823856 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-workload-socket" pod "7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" (UID: "7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:25.824112 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.823979 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-credential-socket" pod "7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" (UID: "7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:25.824314 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.824209 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:25.824314 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.824229 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:25.824314 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.824245 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:25.824314 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.824257 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:25.824314 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.824271 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:25.825710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.825684 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-kube-api-access-dhfdq" pod "7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" (UID: "7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd"). InnerVolumeSpecName "kube-api-access-dhfdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:25.826290 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.826247 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-token" pod "7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" (UID: "7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:25.826290 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.826271 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-envoy" pod "7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" (UID: "7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:25.826812 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.826790 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-podinfo" pod "7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" (UID: "7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:25.925627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.925601 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:25.925627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.925625 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:25.925627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.925634 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:25.926110 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:25.925643 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhfdq\" (UniqueName: \"kubernetes.io/projected/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd-kube-api-access-dhfdq\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:26.012555 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:26.012450 4603 generic.go:356] "Generic (PLEG): container finished" podID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerID="992e118a84566c7df7c84dd5b94ec3cf48e2b89363ad1ff44bddc6c0d9dfa7d4" exitCode=0 Oct 30 18:26:26.012555 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:26.012522 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" Oct 30 18:26:26.012735 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:26.012545 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" event={"ID":"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd","Type":"ContainerDied","Data":"992e118a84566c7df7c84dd5b94ec3cf48e2b89363ad1ff44bddc6c0d9dfa7d4"} Oct 30 18:26:26.012735 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:26.012604 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" event={"ID":"7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd","Type":"ContainerDied","Data":"1e5af9b0ff55f75c72f909e4af6b746403e78714d59449f5456cb7814fc7de5f"} Oct 30 18:26:26.012735 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:26.012626 4603 scope.go:122] "RemoveContainer" containerID="992e118a84566c7df7c84dd5b94ec3cf48e2b89363ad1ff44bddc6c0d9dfa7d4" Oct 30 18:26:26.024098 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:26.024078 4603 scope.go:122] "RemoveContainer" containerID="bb57dd61d380ff4d5f80723b7a4b6801e9620be7bdf496c960b11333446b6337" Oct 30 18:26:26.033882 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:26.033861 4603 scope.go:122] "RemoveContainer" containerID="992e118a84566c7df7c84dd5b94ec3cf48e2b89363ad1ff44bddc6c0d9dfa7d4" Oct 30 18:26:26.034124 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:26:26.034107 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"992e118a84566c7df7c84dd5b94ec3cf48e2b89363ad1ff44bddc6c0d9dfa7d4\": container with ID starting with 992e118a84566c7df7c84dd5b94ec3cf48e2b89363ad1ff44bddc6c0d9dfa7d4 not found: ID does not exist" containerID="992e118a84566c7df7c84dd5b94ec3cf48e2b89363ad1ff44bddc6c0d9dfa7d4" Oct 30 18:26:26.034186 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:26.034132 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"992e118a84566c7df7c84dd5b94ec3cf48e2b89363ad1ff44bddc6c0d9dfa7d4"} err="failed to get container status \"992e118a84566c7df7c84dd5b94ec3cf48e2b89363ad1ff44bddc6c0d9dfa7d4\": rpc error: code = NotFound desc = could not find container \"992e118a84566c7df7c84dd5b94ec3cf48e2b89363ad1ff44bddc6c0d9dfa7d4\": container with ID starting with 992e118a84566c7df7c84dd5b94ec3cf48e2b89363ad1ff44bddc6c0d9dfa7d4 not found: ID does not exist" Oct 30 18:26:26.034186 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:26.034150 4603 scope.go:122] "RemoveContainer" containerID="bb57dd61d380ff4d5f80723b7a4b6801e9620be7bdf496c960b11333446b6337" Oct 30 18:26:26.034404 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:26:26.034383 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb57dd61d380ff4d5f80723b7a4b6801e9620be7bdf496c960b11333446b6337\": container with ID starting with bb57dd61d380ff4d5f80723b7a4b6801e9620be7bdf496c960b11333446b6337 not found: ID does not exist" containerID="bb57dd61d380ff4d5f80723b7a4b6801e9620be7bdf496c960b11333446b6337" Oct 30 18:26:26.034464 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:26.034414 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb57dd61d380ff4d5f80723b7a4b6801e9620be7bdf496c960b11333446b6337"} err="failed to get container status \"bb57dd61d380ff4d5f80723b7a4b6801e9620be7bdf496c960b11333446b6337\": rpc error: code = NotFound desc = could not find container \"bb57dd61d380ff4d5f80723b7a4b6801e9620be7bdf496c960b11333446b6337\": container with ID starting with bb57dd61d380ff4d5f80723b7a4b6801e9620be7bdf496c960b11333446b6337 not found: ID does not exist" Oct 30 18:26:26.080028 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:26.080006 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx"] Oct 30 18:26:26.083464 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:26.083445 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx"] Oct 30 18:26:26.643689 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:26.643659 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:26.701525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:26.698933 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:26:26.701525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:26.699005 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-1fae38-openshift-default-754ffb8497-4ctbx" podUID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:27.410344 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:27.410316 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" path="/var/lib/kubelet/pods/7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd/volumes" Oct 30 18:26:28.438464 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:28.438428 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:29.469923 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.469863 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:26:29.483425 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.483404 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-credential-socket\") pod \"9d3f60a0-c196-48e7-9765-b1f3d8965367\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " Oct 30 18:26:29.483567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.483433 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-envoy\") pod \"9d3f60a0-c196-48e7-9765-b1f3d8965367\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " Oct 30 18:26:29.483567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.483466 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-workload-socket\") pod \"9d3f60a0-c196-48e7-9765-b1f3d8965367\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " Oct 30 18:26:29.483567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.483526 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d3f60a0-c196-48e7-9765-b1f3d8965367-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3f60a0-c196-48e7-9765-b1f3d8965367-istiod-ca-cert\") pod \"9d3f60a0-c196-48e7-9765-b1f3d8965367\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " Oct 30 18:26:29.483756 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.483577 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-workload-certs\") pod \"9d3f60a0-c196-48e7-9765-b1f3d8965367\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " Oct 30 18:26:29.483756 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.483619 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3f60a0-c196-48e7-9765-b1f3d8965367-kube-api-access-lf4dp\" (UniqueName: \"kubernetes.io/projected/9d3f60a0-c196-48e7-9765-b1f3d8965367-kube-api-access-lf4dp\") pod \"9d3f60a0-c196-48e7-9765-b1f3d8965367\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " Oct 30 18:26:29.483910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.483767 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-workload-socket" pod "9d3f60a0-c196-48e7-9765-b1f3d8965367" (UID: "9d3f60a0-c196-48e7-9765-b1f3d8965367"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:29.483910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.483783 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-credential-socket" pod "9d3f60a0-c196-48e7-9765-b1f3d8965367" (UID: "9d3f60a0-c196-48e7-9765-b1f3d8965367"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:29.483910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.483793 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-token\" (UniqueName: \"kubernetes.io/projected/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-token\") pod \"9d3f60a0-c196-48e7-9765-b1f3d8965367\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " Oct 30 18:26:29.483910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.483858 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-workload-certs" pod "9d3f60a0-c196-48e7-9765-b1f3d8965367" (UID: "9d3f60a0-c196-48e7-9765-b1f3d8965367"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:29.484146 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.483916 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d3f60a0-c196-48e7-9765-b1f3d8965367-istiod-ca-cert" pod "9d3f60a0-c196-48e7-9765-b1f3d8965367" (UID: "9d3f60a0-c196-48e7-9765-b1f3d8965367"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:29.484146 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.483914 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-podinfo\") pod \"9d3f60a0-c196-48e7-9765-b1f3d8965367\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " Oct 30 18:26:29.484146 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.483987 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-data\") pod \"9d3f60a0-c196-48e7-9765-b1f3d8965367\" (UID: \"9d3f60a0-c196-48e7-9765-b1f3d8965367\") " Oct 30 18:26:29.484283 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.484256 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-data" pod "9d3f60a0-c196-48e7-9765-b1f3d8965367" (UID: "9d3f60a0-c196-48e7-9765-b1f3d8965367"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:29.485085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.485048 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:29.485085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.485076 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:29.485227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.485091 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:29.485227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.485108 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:29.485227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.485122 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3f60a0-c196-48e7-9765-b1f3d8965367-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:29.486189 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.486163 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-podinfo" pod "9d3f60a0-c196-48e7-9765-b1f3d8965367" (UID: "9d3f60a0-c196-48e7-9765-b1f3d8965367"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:29.486275 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.486203 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3f60a0-c196-48e7-9765-b1f3d8965367-kube-api-access-lf4dp" pod "9d3f60a0-c196-48e7-9765-b1f3d8965367" (UID: "9d3f60a0-c196-48e7-9765-b1f3d8965367"). InnerVolumeSpecName "kube-api-access-lf4dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:29.486395 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.486375 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-envoy" pod "9d3f60a0-c196-48e7-9765-b1f3d8965367" (UID: "9d3f60a0-c196-48e7-9765-b1f3d8965367"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:29.486451 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.486389 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-token" pod "9d3f60a0-c196-48e7-9765-b1f3d8965367" (UID: "9d3f60a0-c196-48e7-9765-b1f3d8965367"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:29.586430 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.586399 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf4dp\" (UniqueName: \"kubernetes.io/projected/9d3f60a0-c196-48e7-9765-b1f3d8965367-kube-api-access-lf4dp\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:29.586430 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.586430 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:29.586648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.586441 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:29.586648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.586450 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3f60a0-c196-48e7-9765-b1f3d8965367-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:29.797628 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.797561 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz"] Oct 30 18:26:29.798211 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.798190 4603 cpu_manager.go:414] "containerMap: removing container" podUID="9d3f60a0-c196-48e7-9765-b1f3d8965367" containerName="istio-proxy" Oct 30 18:26:29.798211 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.798210 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3f60a0-c196-48e7-9765-b1f3d8965367" containerName="istio-proxy" Oct 30 18:26:29.798356 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.798230 4603 cpu_manager.go:414] "containerMap: removing container" podUID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerName="istio-proxy" Oct 30 18:26:29.798356 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.798236 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerName="istio-proxy" Oct 30 18:26:29.798356 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.798245 4603 cpu_manager.go:414] "containerMap: removing container" podUID="9d3f60a0-c196-48e7-9765-b1f3d8965367" containerName="istio-proxy" Oct 30 18:26:29.798356 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.798251 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3f60a0-c196-48e7-9765-b1f3d8965367" containerName="istio-proxy" Oct 30 18:26:29.798356 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.798269 4603 cpu_manager.go:414] "containerMap: removing container" podUID="6109f83b-e952-42b8-bd57-c7f53f8adadb" containerName="maas-api" Oct 30 18:26:29.798356 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.798273 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6109f83b-e952-42b8-bd57-c7f53f8adadb" containerName="maas-api" Oct 30 18:26:29.798741 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.798364 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerName="istio-proxy" Oct 30 18:26:29.798741 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.798372 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3f60a0-c196-48e7-9765-b1f3d8965367" containerName="istio-proxy" Oct 30 18:26:29.798741 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.798382 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="6109f83b-e952-42b8-bd57-c7f53f8adadb" containerName="maas-api" Oct 30 18:26:29.802007 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.801988 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:29.826898 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.826870 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz"] Oct 30 18:26:29.866699 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.866676 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-602e18-openshift-default-dockercfg-nnflx\"" Oct 30 18:26:29.880729 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.880710 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:29.890087 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.890060 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07edd101-b4e3-48b5-95fc-2168b424cfd9-istiod-ca-cert\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:29.890183 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.890117 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-data\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:29.890223 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.890199 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-envoy\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:29.890260 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.890222 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-token\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:29.890260 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.890249 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-podinfo\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:29.890376 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.890363 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-credential-socket\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:29.890420 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.890391 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-workload-socket\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:29.890489 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.890475 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlkbr\" (UniqueName: \"kubernetes.io/projected/07edd101-b4e3-48b5-95fc-2168b424cfd9-kube-api-access-dlkbr\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:29.890586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.890532 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-workload-certs\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:30.001526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.998463 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07edd101-b4e3-48b5-95fc-2168b424cfd9-istiod-ca-cert\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:30.001526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.998591 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-data\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:30.001526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.998647 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-envoy\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:30.001526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.998674 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-token\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:30.001526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.998714 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-podinfo\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:30.001526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.998808 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-credential-socket\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:30.001526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.998847 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-workload-socket\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:30.001526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.998975 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlkbr\" (UniqueName: \"kubernetes.io/projected/07edd101-b4e3-48b5-95fc-2168b424cfd9-kube-api-access-dlkbr\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:30.001526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.999016 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-workload-certs\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:30.001526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:29.999559 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-workload-certs\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:30.001526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.000281 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07edd101-b4e3-48b5-95fc-2168b424cfd9-istiod-ca-cert\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:30.001526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.000569 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-data\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:30.016464 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.016432 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-credential-socket\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:30.024204 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.024175 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-workload-socket\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:30.030747 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.030485 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-podinfo\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:30.073901 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.073822 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlkbr\" (UniqueName: \"kubernetes.io/projected/07edd101-b4e3-48b5-95fc-2168b424cfd9-kube-api-access-dlkbr\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:30.074826 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.074800 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-envoy\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:30.092524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.089004 4603 generic.go:356] "Generic (PLEG): container finished" podID="9d3f60a0-c196-48e7-9765-b1f3d8965367" containerID="8c43fa786feb1975bf4a240378fbd2b573782e07262b74fabca7f0a43454d83c" exitCode=0 Oct 30 18:26:30.092524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.089162 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" event={"ID":"9d3f60a0-c196-48e7-9765-b1f3d8965367","Type":"ContainerDied","Data":"8c43fa786feb1975bf4a240378fbd2b573782e07262b74fabca7f0a43454d83c"} Oct 30 18:26:30.092524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.089226 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" event={"ID":"9d3f60a0-c196-48e7-9765-b1f3d8965367","Type":"ContainerDied","Data":"2471aa6c6e5ef6a986317a5ba2578378470b147124ec5f414c5af62b46a4dcec"} Oct 30 18:26:30.092524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.089261 4603 scope.go:122] "RemoveContainer" containerID="8c43fa786feb1975bf4a240378fbd2b573782e07262b74fabca7f0a43454d83c" Oct 30 18:26:30.092524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.089422 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs" Oct 30 18:26:30.113584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.105067 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-token\") pod \"e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:30.126838 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.126804 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:30.258138 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.249654 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs"] Oct 30 18:26:30.258138 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.249740 4603 scope.go:122] "RemoveContainer" containerID="69dadf26749f14c3df88d4f5b7b25f26514065a46b9b7b9e74baef4efec33e3c" Oct 30 18:26:30.258138 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.252971 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-ca5f33-openshift-default-77f699797c-gdcrs"] Oct 30 18:26:30.283726 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.283699 4603 scope.go:122] "RemoveContainer" containerID="8c43fa786feb1975bf4a240378fbd2b573782e07262b74fabca7f0a43454d83c" Oct 30 18:26:30.284077 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:26:30.284045 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c43fa786feb1975bf4a240378fbd2b573782e07262b74fabca7f0a43454d83c\": container with ID starting with 8c43fa786feb1975bf4a240378fbd2b573782e07262b74fabca7f0a43454d83c not found: ID does not exist" containerID="8c43fa786feb1975bf4a240378fbd2b573782e07262b74fabca7f0a43454d83c" Oct 30 18:26:30.284183 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.284088 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c43fa786feb1975bf4a240378fbd2b573782e07262b74fabca7f0a43454d83c"} err="failed to get container status \"8c43fa786feb1975bf4a240378fbd2b573782e07262b74fabca7f0a43454d83c\": rpc error: code = NotFound desc = could not find container \"8c43fa786feb1975bf4a240378fbd2b573782e07262b74fabca7f0a43454d83c\": container with ID starting with 8c43fa786feb1975bf4a240378fbd2b573782e07262b74fabca7f0a43454d83c not found: ID does not exist" Oct 30 18:26:30.284183 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.284108 4603 scope.go:122] "RemoveContainer" containerID="69dadf26749f14c3df88d4f5b7b25f26514065a46b9b7b9e74baef4efec33e3c" Oct 30 18:26:30.284438 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:26:30.284402 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69dadf26749f14c3df88d4f5b7b25f26514065a46b9b7b9e74baef4efec33e3c\": container with ID starting with 69dadf26749f14c3df88d4f5b7b25f26514065a46b9b7b9e74baef4efec33e3c not found: ID does not exist" containerID="69dadf26749f14c3df88d4f5b7b25f26514065a46b9b7b9e74baef4efec33e3c" Oct 30 18:26:30.284478 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.284445 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69dadf26749f14c3df88d4f5b7b25f26514065a46b9b7b9e74baef4efec33e3c"} err="failed to get container status \"69dadf26749f14c3df88d4f5b7b25f26514065a46b9b7b9e74baef4efec33e3c\": rpc error: code = NotFound desc = could not find container \"69dadf26749f14c3df88d4f5b7b25f26514065a46b9b7b9e74baef4efec33e3c\": container with ID starting with 69dadf26749f14c3df88d4f5b7b25f26514065a46b9b7b9e74baef4efec33e3c not found: ID does not exist" Oct 30 18:26:30.519582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.518529 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:30.618631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.618607 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz"] Oct 30 18:26:30.620877 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:26:30.620848 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07edd101_b4e3_48b5_95fc_2168b424cfd9.slice/crio-57bb1750797f648a322a1b07a51fb24a3b451e0b65a1fadca362628892c3bc40 WatchSource:0}: Error finding container 57bb1750797f648a322a1b07a51fb24a3b451e0b65a1fadca362628892c3bc40: Status 404 returned error can't find the container with id 57bb1750797f648a322a1b07a51fb24a3b451e0b65a1fadca362628892c3bc40 Oct 30 18:26:30.623725 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.623689 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:26:30.623833 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.623772 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:26:30.623833 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.623818 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:26:30.669210 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:30.669168 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:31.101436 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:31.101307 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" event={"ID":"07edd101-b4e3-48b5-95fc-2168b424cfd9","Type":"ContainerStarted","Data":"71a667c6a43e514f3e9d7f5cddf6ed3c161e7ffbcd10947dd4efecaca8a18773"} Oct 30 18:26:31.101745 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:31.101707 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" event={"ID":"07edd101-b4e3-48b5-95fc-2168b424cfd9","Type":"ContainerStarted","Data":"57bb1750797f648a322a1b07a51fb24a3b451e0b65a1fadca362628892c3bc40"} Oct 30 18:26:31.128116 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:31.128082 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:31.133410 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:31.133388 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:31.148052 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:31.148007 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" podStartSLOduration=2.14799475 podStartE2EDuration="2.14799475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:31.146974203 +0000 UTC m=+2051.862544078" watchObservedRunningTime="2026-10-30 18:26:31.14799475 +0000 UTC m=+2051.863564656" Oct 30 18:26:31.416254 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:31.416173 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d3f60a0-c196-48e7-9765-b1f3d8965367" path="/var/lib/kubelet/pods/9d3f60a0-c196-48e7-9765-b1f3d8965367/volumes" Oct 30 18:26:32.108512 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:32.108464 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:32.109224 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:32.109206 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:26:32.680645 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:32.680598 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:33.715699 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:33.715665 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:33.857591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:33.857542 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:34.679155 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:34.679106 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:35.427840 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:35.427788 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:35.564863 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:35.564832 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:36.335410 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:36.335354 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:38.190530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:38.190476 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:39.955087 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:39.955042 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:40.243198 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:40.243115 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m"] Oct 30 18:26:40.278920 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:40.278888 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr"] Oct 30 18:26:40.432742 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:40.432710 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:40.576655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:40.576623 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:42.188529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:42.177640 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:43.857534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:43.852341 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:45.243393 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:45.243355 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m" podUID="9641ab7d-afbe-405b-b4b6-f01f0d403c84" containerName="payload-processing" containerID="cri-o://5a6e86c15c20f6e68ba24fb50dc88d9fecedcac9385d4b27bd1ce19e1a64d157" gracePeriod=25 Oct 30 18:26:45.280056 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:45.280009 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr" podUID="831276c5-adb1-40ab-8bd2-2aa6ec757865" containerName="payload-pre-processing" containerID="cri-o://1e831e77dc7f594d3e9dec2a36a7968760ee3e1a9d162ac4608d6ff22f3284a5" gracePeriod=25 Oct 30 18:26:45.337475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:45.337443 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:45.427529 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:26:45.425719 4603 logging.go:55] [core] [Channel #1315 SubChannel #1316]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.132:50052", ServerName: "10.133.0.132:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.132:50052: connect: connection refused" Oct 30 18:26:45.435726 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:26:45.435699 4603 logging.go:55] [core] [Channel #1317 SubChannel #1318]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.133:50052", ServerName: "10.133.0.133:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.133:50052: connect: connection refused" Oct 30 18:26:45.666053 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:45.666021 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m" Oct 30 18:26:45.699008 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:45.698855 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9641ab7d-afbe-405b-b4b6-f01f0d403c84-kube-api-access-8hlnd\" (UniqueName: \"kubernetes.io/projected/9641ab7d-afbe-405b-b4b6-f01f0d403c84-kube-api-access-8hlnd\") pod \"9641ab7d-afbe-405b-b4b6-f01f0d403c84\" (UID: \"9641ab7d-afbe-405b-b4b6-f01f0d403c84\") " Oct 30 18:26:45.699008 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:45.698931 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9641ab7d-afbe-405b-b4b6-f01f0d403c84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9641ab7d-afbe-405b-b4b6-f01f0d403c84-plugins-config-volume\") pod \"9641ab7d-afbe-405b-b4b6-f01f0d403c84\" (UID: \"9641ab7d-afbe-405b-b4b6-f01f0d403c84\") " Oct 30 18:26:45.699280 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:45.699258 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9641ab7d-afbe-405b-b4b6-f01f0d403c84-plugins-config-volume" pod "9641ab7d-afbe-405b-b4b6-f01f0d403c84" (UID: "9641ab7d-afbe-405b-b4b6-f01f0d403c84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:45.699629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:45.699608 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9641ab7d-afbe-405b-b4b6-f01f0d403c84-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:45.701122 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:45.701101 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9641ab7d-afbe-405b-b4b6-f01f0d403c84-kube-api-access-8hlnd" pod "9641ab7d-afbe-405b-b4b6-f01f0d403c84" (UID: "9641ab7d-afbe-405b-b4b6-f01f0d403c84"). InnerVolumeSpecName "kube-api-access-8hlnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:45.717267 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:45.717242 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr" Oct 30 18:26:45.800972 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:45.800946 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/831276c5-adb1-40ab-8bd2-2aa6ec757865-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/831276c5-adb1-40ab-8bd2-2aa6ec757865-plugins-config-volume\") pod \"831276c5-adb1-40ab-8bd2-2aa6ec757865\" (UID: \"831276c5-adb1-40ab-8bd2-2aa6ec757865\") " Oct 30 18:26:45.801114 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:45.801102 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/831276c5-adb1-40ab-8bd2-2aa6ec757865-kube-api-access-46v68\" (UniqueName: \"kubernetes.io/projected/831276c5-adb1-40ab-8bd2-2aa6ec757865-kube-api-access-46v68\") pod \"831276c5-adb1-40ab-8bd2-2aa6ec757865\" (UID: \"831276c5-adb1-40ab-8bd2-2aa6ec757865\") " Oct 30 18:26:45.801368 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:45.801339 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/831276c5-adb1-40ab-8bd2-2aa6ec757865-plugins-config-volume" pod "831276c5-adb1-40ab-8bd2-2aa6ec757865" (UID: "831276c5-adb1-40ab-8bd2-2aa6ec757865"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:45.801655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:45.801630 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hlnd\" (UniqueName: \"kubernetes.io/projected/9641ab7d-afbe-405b-b4b6-f01f0d403c84-kube-api-access-8hlnd\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:45.801655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:45.801649 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/831276c5-adb1-40ab-8bd2-2aa6ec757865-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:45.803330 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:45.803305 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/831276c5-adb1-40ab-8bd2-2aa6ec757865-kube-api-access-46v68" pod "831276c5-adb1-40ab-8bd2-2aa6ec757865" (UID: "831276c5-adb1-40ab-8bd2-2aa6ec757865"). InnerVolumeSpecName "kube-api-access-46v68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:45.903343 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:45.903317 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46v68\" (UniqueName: \"kubernetes.io/projected/831276c5-adb1-40ab-8bd2-2aa6ec757865-kube-api-access-46v68\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:26:46.326862 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:46.326827 4603 generic.go:356] "Generic (PLEG): container finished" podID="831276c5-adb1-40ab-8bd2-2aa6ec757865" containerID="1e831e77dc7f594d3e9dec2a36a7968760ee3e1a9d162ac4608d6ff22f3284a5" exitCode=0 Oct 30 18:26:46.327317 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:46.326880 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr" Oct 30 18:26:46.327317 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:46.326915 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr" event={"ID":"831276c5-adb1-40ab-8bd2-2aa6ec757865","Type":"ContainerDied","Data":"1e831e77dc7f594d3e9dec2a36a7968760ee3e1a9d162ac4608d6ff22f3284a5"} Oct 30 18:26:46.327317 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:46.327015 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr" event={"ID":"831276c5-adb1-40ab-8bd2-2aa6ec757865","Type":"ContainerDied","Data":"b756216de18049b34733665bd6ff9dd34fe4c4c182a35c9dd84df44dd165c8a3"} Oct 30 18:26:46.327317 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:46.327047 4603 scope.go:122] "RemoveContainer" containerID="1e831e77dc7f594d3e9dec2a36a7968760ee3e1a9d162ac4608d6ff22f3284a5" Oct 30 18:26:46.328536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:46.328491 4603 generic.go:356] "Generic (PLEG): container finished" podID="9641ab7d-afbe-405b-b4b6-f01f0d403c84" containerID="5a6e86c15c20f6e68ba24fb50dc88d9fecedcac9385d4b27bd1ce19e1a64d157" exitCode=0 Oct 30 18:26:46.328650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:46.328540 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m" event={"ID":"9641ab7d-afbe-405b-b4b6-f01f0d403c84","Type":"ContainerDied","Data":"5a6e86c15c20f6e68ba24fb50dc88d9fecedcac9385d4b27bd1ce19e1a64d157"} Oct 30 18:26:46.328650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:46.328577 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m" Oct 30 18:26:46.328650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:46.328594 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m" event={"ID":"9641ab7d-afbe-405b-b4b6-f01f0d403c84","Type":"ContainerDied","Data":"d8852686cf524f7c457d1a1eac8a81dbdd5038ed3466a6d85772c0a5b4d0a06d"} Oct 30 18:26:46.338345 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:46.338295 4603 scope.go:122] "RemoveContainer" containerID="1e831e77dc7f594d3e9dec2a36a7968760ee3e1a9d162ac4608d6ff22f3284a5" Oct 30 18:26:46.338637 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:26:46.338613 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e831e77dc7f594d3e9dec2a36a7968760ee3e1a9d162ac4608d6ff22f3284a5\": container with ID starting with 1e831e77dc7f594d3e9dec2a36a7968760ee3e1a9d162ac4608d6ff22f3284a5 not found: ID does not exist" containerID="1e831e77dc7f594d3e9dec2a36a7968760ee3e1a9d162ac4608d6ff22f3284a5" Oct 30 18:26:46.338729 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:46.338647 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e831e77dc7f594d3e9dec2a36a7968760ee3e1a9d162ac4608d6ff22f3284a5"} err="failed to get container status \"1e831e77dc7f594d3e9dec2a36a7968760ee3e1a9d162ac4608d6ff22f3284a5\": rpc error: code = NotFound desc = could not find container \"1e831e77dc7f594d3e9dec2a36a7968760ee3e1a9d162ac4608d6ff22f3284a5\": container with ID starting with 1e831e77dc7f594d3e9dec2a36a7968760ee3e1a9d162ac4608d6ff22f3284a5 not found: ID does not exist" Oct 30 18:26:46.338729 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:46.338670 4603 scope.go:122] "RemoveContainer" containerID="5a6e86c15c20f6e68ba24fb50dc88d9fecedcac9385d4b27bd1ce19e1a64d157" Oct 30 18:26:46.347932 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:46.347915 4603 scope.go:122] "RemoveContainer" containerID="5a6e86c15c20f6e68ba24fb50dc88d9fecedcac9385d4b27bd1ce19e1a64d157" Oct 30 18:26:46.348221 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:26:46.348196 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a6e86c15c20f6e68ba24fb50dc88d9fecedcac9385d4b27bd1ce19e1a64d157\": container with ID starting with 5a6e86c15c20f6e68ba24fb50dc88d9fecedcac9385d4b27bd1ce19e1a64d157 not found: ID does not exist" containerID="5a6e86c15c20f6e68ba24fb50dc88d9fecedcac9385d4b27bd1ce19e1a64d157" Oct 30 18:26:46.348334 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:46.348226 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a6e86c15c20f6e68ba24fb50dc88d9fecedcac9385d4b27bd1ce19e1a64d157"} err="failed to get container status \"5a6e86c15c20f6e68ba24fb50dc88d9fecedcac9385d4b27bd1ce19e1a64d157\": rpc error: code = NotFound desc = could not find container \"5a6e86c15c20f6e68ba24fb50dc88d9fecedcac9385d4b27bd1ce19e1a64d157\": container with ID starting with 5a6e86c15c20f6e68ba24fb50dc88d9fecedcac9385d4b27bd1ce19e1a64d157 not found: ID does not exist" Oct 30 18:26:46.380856 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:46.380820 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m"] Oct 30 18:26:46.384270 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:46.384244 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m"] Oct 30 18:26:46.416813 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:46.416794 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr"] Oct 30 18:26:46.421401 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:46.421380 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr"] Oct 30 18:26:46.424171 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:46.424147 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c2b103-6dbb5dfd79-tjljr" podUID="831276c5-adb1-40ab-8bd2-2aa6ec757865" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.132:50052\" within 1s: context deadline exceeded" Oct 30 18:26:46.424391 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:46.424362 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-b-c2b103-56bf674948-7bb2m" podUID="9641ab7d-afbe-405b-b4b6-f01f0d403c84" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.133:50052\" within 1s: context deadline exceeded" Oct 30 18:26:47.128451 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:47.128400 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:47.411039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:47.410968 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="831276c5-adb1-40ab-8bd2-2aa6ec757865" path="/var/lib/kubelet/pods/831276c5-adb1-40ab-8bd2-2aa6ec757865/volumes" Oct 30 18:26:47.411399 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:47.411353 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9641ab7d-afbe-405b-b4b6-f01f0d403c84" path="/var/lib/kubelet/pods/9641ab7d-afbe-405b-b4b6-f01f0d403c84/volumes" Oct 30 18:26:49.204197 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:49.204165 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:49.366969 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:49.366936 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:51.077560 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:51.077524 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:51.930773 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:26:51.930737 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:26:52.624889 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:52.624861 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:54.240368 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:54.240317 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:55.678762 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:55.678726 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:56.240518 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:56.240470 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:56.391962 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:56.391925 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:56.555527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:56.555475 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:56.835990 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:56.835908 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:57.010524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:57.004252 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:58.481516 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:58.481460 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:59.088313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:59.088285 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:59.222489 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:59.222456 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:59.376685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:59.376518 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:59.732547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:26:59.732437 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:01.087137 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:01.087102 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:02.754252 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:02.754215 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:02.816541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:02.814322 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:27:03.885716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:03.885682 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:04.029175 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:04.029133 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:04.250341 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:04.250163 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps"] Oct 30 18:27:04.250526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:04.250477 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" podUID="933bdf35-df63-46f1-9cb5-62d484a4ecb0" containerName="istio-proxy" containerID="cri-o://8dad318ed6b99a4b62a398986f843fb07dd5edc6b153a67f34c5c2c6d045fcfd" gracePeriod=30 Oct 30 18:27:04.371566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:04.304292 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:04.374692 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:04.365225 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq"] Oct 30 18:27:04.443531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:04.418573 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs"] Oct 30 18:27:04.450534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:04.444253 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" podUID="2546a5f3-a662-42f8-91a2-0870880c3291" containerName="istio-proxy" containerID="cri-o://c8b35cd2e9b0d1e141eaafe89a64fff170c539a0c384e6bdc5e770e2006dbe32" gracePeriod=30 Oct 30 18:27:04.452088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:04.452043 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" podUID="6b16b70e-9b8b-41ce-b39c-4e3e5654f358" containerName="istio-proxy" containerID="cri-o://b7ad0c3456f2dd1238910d1e5c9fe3ff82c16189f38f0e37cfc28e3bcea8bb9f" gracePeriod=30 Oct 30 18:27:04.966260 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:04.966215 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:05.946544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:05.945257 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:06.128592 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:06.119824 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:06.295529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:06.293434 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:06.952568 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:06.951587 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:07.113528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:07.112729 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:07.159529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:07.138628 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:07.258755 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:07.258636 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:07.425884 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:07.425839 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:07.811194 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:07.810852 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:07.961010 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:07.958637 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:27:07.961010 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:07.959335 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:27:08.108608 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:08.075619 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:27:08.108608 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:08.076052 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:27:08.124540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:08.116683 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:27:08.124540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:08.117147 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:27:08.202529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:08.163339 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": context deadline exceeded" Oct 30 18:27:08.202529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:08.163419 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:27:08.271533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:08.260657 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:08.271533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:08.261145 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:27:08.434590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:08.432010 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:08.486487 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:08.460449 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 30 18:27:08.541542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:08.525603 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:27:08.541755 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:08.538549 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:27:08.688135 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:08.688024 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Oct 30 18:27:08.928125 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:08.928088 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:08.978842 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:08.978729 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:27:09.081164 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:09.081120 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:09.334626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:09.087644 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:09.335443 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:09.335419 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:27:09.335722 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:09.335707 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:27:09.337357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:09.160070 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:27:09.366065 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:09.178590 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:27:09.367595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:09.159983 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:09.368207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:09.368186 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:27:09.750108 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:09.750054 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:27:09.769658 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:09.769605 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:27:09.850043 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:09.281901 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:27:09.985975 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:09.980319 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:27:09.985975 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:09.980428 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" podUID="47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:10.038530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:10.030951 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:10.392531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:10.389568 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:10.488534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:10.410362 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:10.488534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:10.478911 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:27:10.488534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:10.481184 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" podUID="db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:11.008532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:10.984709 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:27:11.008532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:10.987031 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:27:11.008532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:10.987292 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:27:11.008532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:10.987347 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:27:11.008532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:10.987385 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:27:11.043572 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:11.041685 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:27:11.144591 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:27:11.134534 4603 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.699s" Oct 30 18:27:11.144591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:11.135336 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:27:11.229532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:11.203058 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:27:11.259920 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:11.257654 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:27:11.259920 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:11.257726 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" podUID="933bdf35-df63-46f1-9cb5-62d484a4ecb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 18:27:11.354543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:11.315181 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:27:11.354543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:11.315236 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" podUID="2546a5f3-a662-42f8-91a2-0870880c3291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 18:27:11.434041 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:11.411854 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:11.458599 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:11.454868 4603 generic.go:356] "Generic (PLEG): container finished" podID="2546a5f3-a662-42f8-91a2-0870880c3291" containerID="c8b35cd2e9b0d1e141eaafe89a64fff170c539a0c384e6bdc5e770e2006dbe32" exitCode=0 Oct 30 18:27:11.568593 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:27:11.555143 4603 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2546a5f3_a662_42f8_91a2_0870880c3291.slice/crio-conmon-c8b35cd2e9b0d1e141eaafe89a64fff170c539a0c384e6bdc5e770e2006dbe32.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b16b70e_9b8b_41ce_b39c_4e3e5654f358.slice/crio-conmon-b7ad0c3456f2dd1238910d1e5c9fe3ff82c16189f38f0e37cfc28e3bcea8bb9f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933bdf35_df63_46f1_9cb5_62d484a4ecb0.slice/crio-8dad318ed6b99a4b62a398986f843fb07dd5edc6b153a67f34c5c2c6d045fcfd.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933bdf35_df63_46f1_9cb5_62d484a4ecb0.slice/crio-conmon-8dad318ed6b99a4b62a398986f843fb07dd5edc6b153a67f34c5c2c6d045fcfd.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:27:11.611900 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:11.611849 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:27:11.628528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:11.612668 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" podUID="c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:11.912116 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:11.912080 4603 generic.go:356] "Generic (PLEG): container finished" podID="6b16b70e-9b8b-41ce-b39c-4e3e5654f358" containerID="b7ad0c3456f2dd1238910d1e5c9fe3ff82c16189f38f0e37cfc28e3bcea8bb9f" exitCode=0 Oct 30 18:27:11.954531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:11.943103 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" event={"ID":"2546a5f3-a662-42f8-91a2-0870880c3291","Type":"ContainerDied","Data":"c8b35cd2e9b0d1e141eaafe89a64fff170c539a0c384e6bdc5e770e2006dbe32"} Oct 30 18:27:11.988115 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:11.988075 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:12.056685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:11.988564 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:12.056685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:11.996540 4603 scope.go:122] "RemoveContainer" containerID="80167c0ce897f4f7e70aab6ec21b6448441828671060fbd79488eb94d2d07f16" Oct 30 18:27:12.056685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:12.003669 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" event={"ID":"6b16b70e-9b8b-41ce-b39c-4e3e5654f358","Type":"ContainerDied","Data":"b7ad0c3456f2dd1238910d1e5c9fe3ff82c16189f38f0e37cfc28e3bcea8bb9f"} Oct 30 18:27:12.097536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:12.095920 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:12.139698 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:12.139646 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:12.145541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:12.144609 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:12.174582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:12.145804 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:12.174582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:12.146883 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:27:12.234861 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:12.227457 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:27:12.466588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:12.413556 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:12.466588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:12.414564 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:13.075581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:13.053666 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:13.075581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:13.056452 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:13.155929 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:13.155878 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:27:13.156785 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:13.156750 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:13.157255 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:13.157226 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:13.180809 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:13.157629 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:27:13.257762 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:13.229568 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:27:13.401269 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:13.401165 4603 generic.go:356] "Generic (PLEG): container finished" podID="933bdf35-df63-46f1-9cb5-62d484a4ecb0" containerID="8dad318ed6b99a4b62a398986f843fb07dd5edc6b153a67f34c5c2c6d045fcfd" exitCode=0 Oct 30 18:27:13.401706 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:13.401662 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" event={"ID":"933bdf35-df63-46f1-9cb5-62d484a4ecb0","Type":"ContainerDied","Data":"8dad318ed6b99a4b62a398986f843fb07dd5edc6b153a67f34c5c2c6d045fcfd"} Oct 30 18:27:13.415680 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:13.415601 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:27:13.584730 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:13.581182 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:27:13.584730 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:13.581237 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" podUID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:13.992792 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:13.948630 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:27:14.061183 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.061028 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:27:14.072321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.061430 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:27:14.072321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.064676 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-data\") pod \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " Oct 30 18:27:14.072321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.064721 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istiod-ca-cert\") pod \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " Oct 30 18:27:14.072321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.064796 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-credential-socket\") pod \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " Oct 30 18:27:14.072321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.064825 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-workload-socket\") pod \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " Oct 30 18:27:14.072321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.064964 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-podinfo\") pod \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " Oct 30 18:27:14.072321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.065028 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-kube-api-access-rjkhw\" (UniqueName: \"kubernetes.io/projected/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-kube-api-access-rjkhw\") pod \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " Oct 30 18:27:14.072321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.065069 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-workload-certs\") pod \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " Oct 30 18:27:14.072321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.065137 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-token\" (UniqueName: \"kubernetes.io/projected/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-token\") pod \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " Oct 30 18:27:14.072321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.065291 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-envoy\") pod \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\" (UID: \"6b16b70e-9b8b-41ce-b39c-4e3e5654f358\") " Oct 30 18:27:14.073341 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.073317 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-workload-socket" pod "6b16b70e-9b8b-41ce-b39c-4e3e5654f358" (UID: "6b16b70e-9b8b-41ce-b39c-4e3e5654f358"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:14.073716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.073697 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-data" pod "6b16b70e-9b8b-41ce-b39c-4e3e5654f358" (UID: "6b16b70e-9b8b-41ce-b39c-4e3e5654f358"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:14.074133 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.074115 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istiod-ca-cert" pod "6b16b70e-9b8b-41ce-b39c-4e3e5654f358" (UID: "6b16b70e-9b8b-41ce-b39c-4e3e5654f358"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:14.074469 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.074453 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-credential-socket" pod "6b16b70e-9b8b-41ce-b39c-4e3e5654f358" (UID: "6b16b70e-9b8b-41ce-b39c-4e3e5654f358"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:14.106217 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.106183 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-workload-certs" pod "6b16b70e-9b8b-41ce-b39c-4e3e5654f358" (UID: "6b16b70e-9b8b-41ce-b39c-4e3e5654f358"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:14.125962 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.125923 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:27:14.126154 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.126001 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" podUID="22e0b280-2b65-4fad-8344-67049190d108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:14.147249 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.147219 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:27:14.193542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.168316 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:14.193542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.168688 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:27:14.193542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.168961 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:27:14.193542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.171387 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.193542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.171410 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.193542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.171425 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.193542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.171437 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.193542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.171450 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.217535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.201755 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:14.234021 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.233960 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-podinfo" pod "6b16b70e-9b8b-41ce-b39c-4e3e5654f358" (UID: "6b16b70e-9b8b-41ce-b39c-4e3e5654f358"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:27:14.262931 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.262541 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:14.285549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.285394 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2546a5f3-a662-42f8-91a2-0870880c3291-kube-api-access-kr46s\" (UniqueName: \"kubernetes.io/projected/2546a5f3-a662-42f8-91a2-0870880c3291-kube-api-access-kr46s\") pod \"2546a5f3-a662-42f8-91a2-0870880c3291\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " Oct 30 18:27:14.285549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.285522 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-istio-envoy\") pod \"2546a5f3-a662-42f8-91a2-0870880c3291\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " Oct 30 18:27:14.285809 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.285686 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2546a5f3-a662-42f8-91a2-0870880c3291-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2546a5f3-a662-42f8-91a2-0870880c3291-istio-podinfo\") pod \"2546a5f3-a662-42f8-91a2-0870880c3291\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " Oct 30 18:27:14.285809 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.285764 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2546a5f3-a662-42f8-91a2-0870880c3291-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2546a5f3-a662-42f8-91a2-0870880c3291-istiod-ca-cert\") pod \"2546a5f3-a662-42f8-91a2-0870880c3291\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " Oct 30 18:27:14.285925 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.285812 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-istio-data\") pod \"2546a5f3-a662-42f8-91a2-0870880c3291\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " Oct 30 18:27:14.297562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.289641 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-workload-certs\") pod \"2546a5f3-a662-42f8-91a2-0870880c3291\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " Oct 30 18:27:14.316156 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.289743 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-workload-socket\") pod \"2546a5f3-a662-42f8-91a2-0870880c3291\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " Oct 30 18:27:14.316156 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.312812 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-credential-socket\") pod \"2546a5f3-a662-42f8-91a2-0870880c3291\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " Oct 30 18:27:14.316156 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.312944 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2546a5f3-a662-42f8-91a2-0870880c3291-istio-token\" (UniqueName: \"kubernetes.io/projected/2546a5f3-a662-42f8-91a2-0870880c3291-istio-token\") pod \"2546a5f3-a662-42f8-91a2-0870880c3291\" (UID: \"2546a5f3-a662-42f8-91a2-0870880c3291\") " Oct 30 18:27:14.324806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.324074 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.399530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.393674 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-credential-socket" pod "2546a5f3-a662-42f8-91a2-0870880c3291" (UID: "2546a5f3-a662-42f8-91a2-0870880c3291"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:14.469550 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.420846 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-workload-certs" pod "2546a5f3-a662-42f8-91a2-0870880c3291" (UID: "2546a5f3-a662-42f8-91a2-0870880c3291"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:14.469550 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.424578 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-istio-data" pod "2546a5f3-a662-42f8-91a2-0870880c3291" (UID: "2546a5f3-a662-42f8-91a2-0870880c3291"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:14.469550 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.424657 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:14.469550 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.430581 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-workload-socket" pod "2546a5f3-a662-42f8-91a2-0870880c3291" (UID: "2546a5f3-a662-42f8-91a2-0870880c3291"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:14.469550 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.431220 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:27:14.469550 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.431266 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" podUID="5411300d-c1aa-4397-be3d-0ce869a1c1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:14.469550 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.440884 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2546a5f3-a662-42f8-91a2-0870880c3291-istiod-ca-cert" pod "2546a5f3-a662-42f8-91a2-0870880c3291" (UID: "2546a5f3-a662-42f8-91a2-0870880c3291"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:14.515421 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.474233 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2546a5f3-a662-42f8-91a2-0870880c3291-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.515421 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.474267 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.515421 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.474284 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.515421 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.474301 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.515421 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.474315 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.654074 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.618352 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Oct 30 18:27:14.868530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.842877 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:27:14.868530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:14.842938 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" podUID="81f49829-73ed-4760-9444-f1793861eba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:15.039530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.027541 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2546a5f3-a662-42f8-91a2-0870880c3291-istio-token" pod "2546a5f3-a662-42f8-91a2-0870880c3291" (UID: "2546a5f3-a662-42f8-91a2-0870880c3291"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:15.064665 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.064622 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:27:15.083529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.068580 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" podUID="571fc9dd-71e3-4fa7-b1af-e74f0a96654d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:15.083855 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.072671 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2546a5f3-a662-42f8-91a2-0870880c3291-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:15.117479 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.090113 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:27:15.117479 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.090575 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:15.117479 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.105714 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-token" pod "6b16b70e-9b8b-41ce-b39c-4e3e5654f358" (UID: "6b16b70e-9b8b-41ce-b39c-4e3e5654f358"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:15.117479 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.109582 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:27:15.148765 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.142925 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-envoy" pod "6b16b70e-9b8b-41ce-b39c-4e3e5654f358" (UID: "6b16b70e-9b8b-41ce-b39c-4e3e5654f358"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:15.157581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.155240 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2546a5f3-a662-42f8-91a2-0870880c3291-kube-api-access-kr46s" pod "2546a5f3-a662-42f8-91a2-0870880c3291" (UID: "2546a5f3-a662-42f8-91a2-0870880c3291"). InnerVolumeSpecName "kube-api-access-kr46s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:15.157581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.155487 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2546a5f3-a662-42f8-91a2-0870880c3291-istio-podinfo" pod "2546a5f3-a662-42f8-91a2-0870880c3291" (UID: "2546a5f3-a662-42f8-91a2-0870880c3291"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.168867 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.169391 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq" event={"ID":"6b16b70e-9b8b-41ce-b39c-4e3e5654f358","Type":"ContainerDied","Data":"8c06981f9a5bee9e77dcb2ecd7f8c9fa96eb0dd39a166ed6bb3027df8dcb709f"} Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.169452 4603 scope.go:122] "RemoveContainer" containerID="b7ad0c3456f2dd1238910d1e5c9fe3ff82c16189f38f0e37cfc28e3bcea8bb9f" Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.170653 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.170919 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.171091 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.186164 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-envoy\") pod \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.186244 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/933bdf35-df63-46f1-9cb5-62d484a4ecb0-kube-api-access-6xkw4\" (UniqueName: \"kubernetes.io/projected/933bdf35-df63-46f1-9cb5-62d484a4ecb0-kube-api-access-6xkw4\") pod \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.186322 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-token\" (UniqueName: \"kubernetes.io/projected/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-token\") pod \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.186440 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-workload-socket\") pod \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.186564 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istiod-ca-cert\") pod \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.186781 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-workload-certs\") pod \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.186817 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-podinfo\") pod \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.186879 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-data\") pod \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.186992 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-credential-socket\") pod \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\" (UID: \"933bdf35-df63-46f1-9cb5-62d484a4ecb0\") " Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.191012 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.191035 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.191048 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr46s\" (UniqueName: \"kubernetes.io/projected/2546a5f3-a662-42f8-91a2-0870880c3291-kube-api-access-kr46s\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.191076 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2546a5f3-a662-42f8-91a2-0870880c3291-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.195590 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-workload-socket" pod "933bdf35-df63-46f1-9cb5-62d484a4ecb0" (UID: "933bdf35-df63-46f1-9cb5-62d484a4ecb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.212713 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-data" pod "933bdf35-df63-46f1-9cb5-62d484a4ecb0" (UID: "933bdf35-df63-46f1-9cb5-62d484a4ecb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.212795 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.219483 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istiod-ca-cert" pod "933bdf35-df63-46f1-9cb5-62d484a4ecb0" (UID: "933bdf35-df63-46f1-9cb5-62d484a4ecb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.219583 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-workload-certs" pod "933bdf35-df63-46f1-9cb5-62d484a4ecb0" (UID: "933bdf35-df63-46f1-9cb5-62d484a4ecb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:15.170580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.228355 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-credential-socket" pod "933bdf35-df63-46f1-9cb5-62d484a4ecb0" (UID: "933bdf35-df63-46f1-9cb5-62d484a4ecb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:15.352457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.239675 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" event={"ID":"933bdf35-df63-46f1-9cb5-62d484a4ecb0","Type":"ContainerDied","Data":"7fdeb3ad4b7042af2400ed025bdf7f20e3511bd967a846da3b39100dd7bfd7bb"} Oct 30 18:27:15.352457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.239801 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps" Oct 30 18:27:15.352457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.243145 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" event={"ID":"2546a5f3-a662-42f8-91a2-0870880c3291","Type":"ContainerDied","Data":"3cecea1b55e81dc255845d3c48ddcb6a4532d92ca2e1925ecf20546d3224d890"} Oct 30 18:27:15.352457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.243251 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs" Oct 30 18:27:15.352457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.272463 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:15.352457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.283739 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-istio-envoy" pod "2546a5f3-a662-42f8-91a2-0870880c3291" (UID: "2546a5f3-a662-42f8-91a2-0870880c3291"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:15.352457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.305570 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:15.352457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.306552 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:15.352457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.306581 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:15.352457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.306595 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:15.352457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.309902 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:15.352457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.309922 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2546a5f3-a662-42f8-91a2-0870880c3291-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:15.358269 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.358196 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-envoy" pod "933bdf35-df63-46f1-9cb5-62d484a4ecb0" (UID: "933bdf35-df63-46f1-9cb5-62d484a4ecb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:15.846397 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.565173 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933bdf35-df63-46f1-9cb5-62d484a4ecb0-kube-api-access-6xkw4" pod "933bdf35-df63-46f1-9cb5-62d484a4ecb0" (UID: "933bdf35-df63-46f1-9cb5-62d484a4ecb0"). InnerVolumeSpecName "kube-api-access-6xkw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:15.846397 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.565341 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-token" pod "933bdf35-df63-46f1-9cb5-62d484a4ecb0" (UID: "933bdf35-df63-46f1-9cb5-62d484a4ecb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:15.846397 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.565412 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-kube-api-access-rjkhw" pod "6b16b70e-9b8b-41ce-b39c-4e3e5654f358" (UID: "6b16b70e-9b8b-41ce-b39c-4e3e5654f358"). InnerVolumeSpecName "kube-api-access-rjkhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:15.846397 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.565823 4603 scope.go:122] "RemoveContainer" containerID="8dad318ed6b99a4b62a398986f843fb07dd5edc6b153a67f34c5c2c6d045fcfd" Oct 30 18:27:15.846397 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.569697 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Oct 30 18:27:15.846397 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.572881 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-podinfo" pod "933bdf35-df63-46f1-9cb5-62d484a4ecb0" (UID: "933bdf35-df63-46f1-9cb5-62d484a4ecb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:27:15.846397 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.667336 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:15.846397 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.667378 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xkw4\" (UniqueName: \"kubernetes.io/projected/933bdf35-df63-46f1-9cb5-62d484a4ecb0-kube-api-access-6xkw4\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:15.846397 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.667399 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:15.846397 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.667415 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjkhw\" (UniqueName: \"kubernetes.io/projected/6b16b70e-9b8b-41ce-b39c-4e3e5654f358-kube-api-access-rjkhw\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:15.846397 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.667430 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933bdf35-df63-46f1-9cb5-62d484a4ecb0-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:27:15.846397 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.687324 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:27:15.943858 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:15.929873 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:16.108592 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:16.091111 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:16.124528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:16.119827 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:27:16.226557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:16.222641 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:16.444635 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:16.394905 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:16.444635 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:16.425608 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:27:16.508251 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:16.508211 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:27:16.509835 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:16.509801 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:27:16.743582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:16.730059 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:16.792388 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:16.792359 4603 scope.go:122] "RemoveContainer" containerID="c8b35cd2e9b0d1e141eaafe89a64fff170c539a0c384e6bdc5e770e2006dbe32" Oct 30 18:27:16.956294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:16.956263 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq"] Oct 30 18:27:17.057775 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.057740 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:17.085254 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.085219 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-8ncgq"] Oct 30 18:27:17.096114 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.095593 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:17.132524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.122346 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:17.152591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.152551 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs"] Oct 30 18:27:17.174600 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.174555 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:17.222308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.222280 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:27:17.289906 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.231876 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:27:17.423192 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.423155 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:17.424720 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.424283 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded" Oct 30 18:27:17.426233 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.426057 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:17.452759 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.449587 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:17.552593 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.527405 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:17.719587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.711068 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:17.732569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.730605 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:17.752173 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.752126 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b16b70e-9b8b-41ce-b39c-4e3e5654f358" path="/var/lib/kubelet/pods/6b16b70e-9b8b-41ce-b39c-4e3e5654f358/volumes" Oct 30 18:27:17.752988 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.752959 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:27:17.753628 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.753611 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-kvggs"] Oct 30 18:27:17.753812 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.753801 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps"] Oct 30 18:27:17.753960 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.753950 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:27:17.754086 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.754076 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-ckkps"] Oct 30 18:27:17.754221 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.754212 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:27:17.754372 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.754355 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:27:17.848036 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.848008 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:27:17.854234 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.854205 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:27:17.857596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:17.857574 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:27:19.083019 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:19.082984 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:19.464542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:19.464435 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2546a5f3-a662-42f8-91a2-0870880c3291" path="/var/lib/kubelet/pods/2546a5f3-a662-42f8-91a2-0870880c3291/volumes" Oct 30 18:27:19.465227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:19.465208 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="933bdf35-df63-46f1-9cb5-62d484a4ecb0" path="/var/lib/kubelet/pods/933bdf35-df63-46f1-9cb5-62d484a4ecb0/volumes" Oct 30 18:27:20.652638 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:20.652613 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 18:27:20.947679 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:20.947610 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 18:27:21.246810 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:21.246709 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:21.938343 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:27:21.938294 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:27:22.566454 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:22.566413 4603 scope.go:122] "RemoveContainer" containerID="427038c55cc4abe29cb9b28cdcd5ce3cedc61434760160b3b1384de68cdad8dc" Oct 30 18:27:22.807224 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:22.807188 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:24.602024 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:24.601988 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:26.123264 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:26.123228 4603 generic.go:356] "Generic (PLEG): container finished" podID="07edd101-b4e3-48b5-95fc-2168b424cfd9" containerID="71a667c6a43e514f3e9d7f5cddf6ed3c161e7ffbcd10947dd4efecaca8a18773" exitCode=0 Oct 30 18:27:26.123264 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:26.123253 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" event={"ID":"07edd101-b4e3-48b5-95fc-2168b424cfd9","Type":"ContainerDied","Data":"71a667c6a43e514f3e9d7f5cddf6ed3c161e7ffbcd10947dd4efecaca8a18773"} Oct 30 18:27:26.123884 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:26.123686 4603 scope.go:122] "RemoveContainer" containerID="71a667c6a43e514f3e9d7f5cddf6ed3c161e7ffbcd10947dd4efecaca8a18773" Oct 30 18:27:26.125489 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:26.125457 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:27:26.125589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:26.125564 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:27:26.125647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:26.125612 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:27:26.199403 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:26.199377 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:27.051220 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.051187 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:27.132679 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.132632 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" event={"ID":"07edd101-b4e3-48b5-95fc-2168b424cfd9","Type":"ContainerStarted","Data":"ade79797f9d0448cb267c8c6469108edeebf906ca930dfb8176652d30b8d842f"} Oct 30 18:27:27.458683 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.458581 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:27.618583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.616836 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:27.806525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.805516 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk"] Oct 30 18:27:27.816538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.815209 4603 cpu_manager.go:414] "containerMap: removing container" podUID="933bdf35-df63-46f1-9cb5-62d484a4ecb0" containerName="istio-proxy" Oct 30 18:27:27.816538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.815336 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="933bdf35-df63-46f1-9cb5-62d484a4ecb0" containerName="istio-proxy" Oct 30 18:27:27.816538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.815365 4603 cpu_manager.go:414] "containerMap: removing container" podUID="2546a5f3-a662-42f8-91a2-0870880c3291" containerName="istio-proxy" Oct 30 18:27:27.816538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.815405 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2546a5f3-a662-42f8-91a2-0870880c3291" containerName="istio-proxy" Oct 30 18:27:27.816538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.815426 4603 cpu_manager.go:414] "containerMap: removing container" podUID="9641ab7d-afbe-405b-b4b6-f01f0d403c84" containerName="payload-processing" Oct 30 18:27:27.816538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.815441 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9641ab7d-afbe-405b-b4b6-f01f0d403c84" containerName="payload-processing" Oct 30 18:27:27.816538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.815457 4603 cpu_manager.go:414] "containerMap: removing container" podUID="831276c5-adb1-40ab-8bd2-2aa6ec757865" containerName="payload-pre-processing" Oct 30 18:27:27.816538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.815465 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="831276c5-adb1-40ab-8bd2-2aa6ec757865" containerName="payload-pre-processing" Oct 30 18:27:27.816538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.815524 4603 cpu_manager.go:414] "containerMap: removing container" podUID="6b16b70e-9b8b-41ce-b39c-4e3e5654f358" containerName="istio-proxy" Oct 30 18:27:27.816538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.815532 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b16b70e-9b8b-41ce-b39c-4e3e5654f358" containerName="istio-proxy" Oct 30 18:27:27.816538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.815554 4603 cpu_manager.go:414] "containerMap: removing container" podUID="2546a5f3-a662-42f8-91a2-0870880c3291" containerName="istio-proxy" Oct 30 18:27:27.816538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.815563 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2546a5f3-a662-42f8-91a2-0870880c3291" containerName="istio-proxy" Oct 30 18:27:27.816538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.815587 4603 cpu_manager.go:414] "containerMap: removing container" podUID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerName="istio-proxy" Oct 30 18:27:27.816538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.815594 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerName="istio-proxy" Oct 30 18:27:27.816538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.815765 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3f60a0-c196-48e7-9765-b1f3d8965367" containerName="istio-proxy" Oct 30 18:27:27.816538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.815781 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="933bdf35-df63-46f1-9cb5-62d484a4ecb0" containerName="istio-proxy" Oct 30 18:27:27.816538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.815790 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="2546a5f3-a662-42f8-91a2-0870880c3291" containerName="istio-proxy" Oct 30 18:27:27.816538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.815801 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="9641ab7d-afbe-405b-b4b6-f01f0d403c84" containerName="payload-processing" Oct 30 18:27:27.816538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.815811 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="831276c5-adb1-40ab-8bd2-2aa6ec757865" containerName="payload-pre-processing" Oct 30 18:27:27.816538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.815824 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b16b70e-9b8b-41ce-b39c-4e3e5654f358" containerName="istio-proxy" Oct 30 18:27:27.816538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.815836 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="7084c5ce-dafc-4c82-bc60-c9ceccb1cdfd" containerName="istio-proxy" Oct 30 18:27:27.841527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.837919 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk" Oct 30 18:27:27.851539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.850244 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk"] Oct 30 18:27:27.865524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.863762 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-602e18-dockercfg-lmsfz\"" Oct 30 18:27:27.865524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.863902 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-602e18\"" Oct 30 18:27:27.887548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.883871 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591b0147-7c0a-4295-886d-9d238a646b9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk\" (UID: \"591b0147-7c0a-4295-886d-9d238a646b9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk" Oct 30 18:27:27.887548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.883953 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jh7f\" (UniqueName: \"kubernetes.io/projected/591b0147-7c0a-4295-886d-9d238a646b9a-kube-api-access-2jh7f\") pod \"payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk\" (UID: \"591b0147-7c0a-4295-886d-9d238a646b9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk" Oct 30 18:27:27.994952 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.993582 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9"] Oct 30 18:27:27.997055 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:27.995827 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="2546a5f3-a662-42f8-91a2-0870880c3291" containerName="istio-proxy" Oct 30 18:27:28.023313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:28.023131 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591b0147-7c0a-4295-886d-9d238a646b9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk\" (UID: \"591b0147-7c0a-4295-886d-9d238a646b9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk" Oct 30 18:27:28.024794 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:28.024755 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jh7f\" (UniqueName: \"kubernetes.io/projected/591b0147-7c0a-4295-886d-9d238a646b9a-kube-api-access-2jh7f\") pod \"payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk\" (UID: \"591b0147-7c0a-4295-886d-9d238a646b9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk" Oct 30 18:27:28.026639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:28.026617 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591b0147-7c0a-4295-886d-9d238a646b9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk\" (UID: \"591b0147-7c0a-4295-886d-9d238a646b9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk" Oct 30 18:27:28.087469 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:28.053362 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9" Oct 30 18:27:28.087469 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:28.085137 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9"] Oct 30 18:27:28.088268 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:28.088233 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jh7f\" (UniqueName: \"kubernetes.io/projected/591b0147-7c0a-4295-886d-9d238a646b9a-kube-api-access-2jh7f\") pod \"payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk\" (UID: \"591b0147-7c0a-4295-886d-9d238a646b9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk" Oct 30 18:27:28.127716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:28.127676 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b79b5853-8745-436a-aeff-aada86fb4853-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9\" (UID: \"b79b5853-8745-436a-aeff-aada86fb4853\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9" Oct 30 18:27:28.127902 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:28.127860 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrbvj\" (UniqueName: \"kubernetes.io/projected/b79b5853-8745-436a-aeff-aada86fb4853-kube-api-access-lrbvj\") pod \"payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9\" (UID: \"b79b5853-8745-436a-aeff-aada86fb4853\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9" Oct 30 18:27:28.128251 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:28.128231 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:27:28.129655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:28.129628 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 18:27:28.129832 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:28.129711 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" podUID="07edd101-b4e3-48b5-95fc-2168b424cfd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 18:27:28.178167 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:28.178144 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk" Oct 30 18:27:28.229596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:28.229569 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrbvj\" (UniqueName: \"kubernetes.io/projected/b79b5853-8745-436a-aeff-aada86fb4853-kube-api-access-lrbvj\") pod \"payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9\" (UID: \"b79b5853-8745-436a-aeff-aada86fb4853\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9" Oct 30 18:27:28.229857 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:28.229832 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b79b5853-8745-436a-aeff-aada86fb4853-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9\" (UID: \"b79b5853-8745-436a-aeff-aada86fb4853\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9" Oct 30 18:27:28.230488 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:28.230467 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b79b5853-8745-436a-aeff-aada86fb4853-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9\" (UID: \"b79b5853-8745-436a-aeff-aada86fb4853\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9" Oct 30 18:27:28.340024 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:28.281463 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrbvj\" (UniqueName: \"kubernetes.io/projected/b79b5853-8745-436a-aeff-aada86fb4853-kube-api-access-lrbvj\") pod \"payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9\" (UID: \"b79b5853-8745-436a-aeff-aada86fb4853\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9" Oct 30 18:27:28.443666 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:28.443431 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9" Oct 30 18:27:28.574673 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:28.574638 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk"] Oct 30 18:27:28.576047 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:27:28.576015 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591b0147_7c0a_4295_886d_9d238a646b9a.slice/crio-17b94222a287aebc1f30809bdaa9256b06c5656cb6da290386dda57fcd9caff4 WatchSource:0}: Error finding container 17b94222a287aebc1f30809bdaa9256b06c5656cb6da290386dda57fcd9caff4: Status 404 returned error can't find the container with id 17b94222a287aebc1f30809bdaa9256b06c5656cb6da290386dda57fcd9caff4 Oct 30 18:27:28.744598 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:28.744569 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:28.819018 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:28.818932 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9"] Oct 30 18:27:29.129403 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:29.128948 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 18:27:29.129403 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:29.129007 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" podUID="07edd101-b4e3-48b5-95fc-2168b424cfd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 18:27:29.217584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:29.210030 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk" event={"ID":"591b0147-7c0a-4295-886d-9d238a646b9a","Type":"ContainerStarted","Data":"19a20d6097bc953ffa521639253cfb183315b904328bac6c7e403f00354d5a1d"} Oct 30 18:27:29.217584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:29.210111 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk" event={"ID":"591b0147-7c0a-4295-886d-9d238a646b9a","Type":"ContainerStarted","Data":"17b94222a287aebc1f30809bdaa9256b06c5656cb6da290386dda57fcd9caff4"} Oct 30 18:27:29.217584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:29.211296 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk" Oct 30 18:27:29.245526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:29.245468 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9" event={"ID":"b79b5853-8745-436a-aeff-aada86fb4853","Type":"ContainerStarted","Data":"4f0b6dcca8452cf73c69505ae2b28df2f25391a129e8877818f12595f4dc6bcf"} Oct 30 18:27:29.245900 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:29.245864 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9" event={"ID":"b79b5853-8745-436a-aeff-aada86fb4853","Type":"ContainerStarted","Data":"c296c6af29051f4c2275cf6b7c3ad8666a7b35bd510ceac371e816765309dfdd"} Oct 30 18:27:29.280569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:29.277724 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9" Oct 30 18:27:29.289564 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:29.289135 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk" podStartSLOduration=2.289112351 podStartE2EDuration="2.289112351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:29.278096093 +0000 UTC m=+2109.993665968" watchObservedRunningTime="2026-10-30 18:27:29.289112351 +0000 UTC m=+2110.004682229" Oct 30 18:27:30.127687 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:30.127648 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 18:27:30.127913 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:30.127717 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" podUID="07edd101-b4e3-48b5-95fc-2168b424cfd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 18:27:30.421129 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:30.421025 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:31.128226 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:31.128171 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 18:27:31.128397 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:31.128249 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" podUID="07edd101-b4e3-48b5-95fc-2168b424cfd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 18:27:32.044822 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:32.044783 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:32.112102 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:32.111660 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:27:32.132526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:32.129134 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 18:27:32.132526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:32.129240 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" podUID="07edd101-b4e3-48b5-95fc-2168b424cfd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 18:27:33.127776 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:33.127735 4603 patch_prober.go:29] interesting pod/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 18:27:33.128286 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:33.127816 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" podUID="07edd101-b4e3-48b5-95fc-2168b424cfd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 18:27:33.779100 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:33.779062 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:34.132048 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:34.131967 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:27:34.133115 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:34.133098 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:27:34.190603 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:34.190552 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9" podStartSLOduration=7.190539796 podStartE2EDuration="7.190539796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:29.326866226 +0000 UTC m=+2110.042436102" watchObservedRunningTime="2026-10-30 18:27:34.190539796 +0000 UTC m=+2114.906109647" Oct 30 18:27:36.026818 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:36.026785 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:37.690627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:37.686268 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:39.377302 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:39.377270 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:40.254671 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:40.254643 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9" Oct 30 18:27:41.143239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:41.143206 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:41.267663 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:41.267628 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk" Oct 30 18:27:43.420594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:43.420563 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:45.085605 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:45.080048 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:46.615967 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:46.615916 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:48.315111 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:48.315078 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:50.554985 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:50.554944 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:51.930249 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:27:51.930213 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:27:52.236121 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:52.236027 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:53.958196 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:53.958164 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:55.706341 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:55.706298 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:57.731114 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:57.731077 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:59.482326 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:27:59.482284 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:01.335653 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:01.335619 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:03.166606 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:03.166570 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:05.403571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:05.400060 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:07.047471 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:07.047436 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:08.755714 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:08.755679 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:10.495360 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:10.495331 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:12.715737 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:12.715705 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:14.337518 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:14.337463 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:14.488961 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:14.488923 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:15.974779 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:15.974745 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:17.153782 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:17.153749 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:17.651591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:17.651550 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:19.747739 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:19.747682 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:21.033910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:21.033876 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:28:21.557153 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:21.557124 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:21.940932 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:28:21.940843 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:28:24.005529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:24.005008 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:24.604532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:24.603621 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:24.998684 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:24.998587 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:25.037655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:25.037612 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:25.053710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:25.053646 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:25.121614 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:25.121568 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:25.149530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:25.148644 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:25.408629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:25.408582 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:25.521527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:25.520446 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:26.003585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:25.999613 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:26.003585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:26.002643 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:28:26.048709 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:26.048653 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:26.125492 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:26.052733 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:28:26.129719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:26.122736 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:26.154206 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:26.056586 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:26.154206 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:26.151999 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:26.154206 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:26.153152 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:26.181221 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:26.154686 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:28:26.181221 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:26.154755 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:28:26.181221 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:26.154796 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:28:26.181221 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:26.154834 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:28:26.610530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:26.609343 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:28:26.771526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:26.747033 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:28:27.042835 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:27.041773 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:28:27.101544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:27.087787 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:27.155569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:27.146534 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:27.155569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:27.147055 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:28:27.181530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:27.156117 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:27.205530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:27.176216 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:28:27.205530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:27.176404 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:28:27.236524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:27.236348 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:28:27.331012 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:27.330913 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Oct 30 18:28:27.512579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:27.505222 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:28:27.794078 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:27.629817 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:27.895308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:27.629864 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded" Oct 30 18:28:28.043274 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:28.043225 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:28.097823 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:28.097707 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:28.214530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:28.213621 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:28:28.214530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:28.214047 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:28:28.214530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:28.214333 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:28:28.246529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:28.245617 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:28.342534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:28.316154 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": context deadline exceeded" Oct 30 18:28:28.374818 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:28.374675 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:28:28.375157 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:28.375125 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" podUID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:28.672274 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:28.664175 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:28.672274 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:28.664831 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:28.827535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:28.800741 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:28.902531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:28.900606 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:29.051641 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:29.051587 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:28:29.099641 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:29.099595 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:29.240532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:29.229609 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:29.240532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:29.230021 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:28:29.240532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:29.230321 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:28:29.248525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:29.247823 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:29.807701 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:29.804675 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:29.818644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:29.818604 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:29.901650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:29.901602 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:29.921745 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:29.915872 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:28:29.921745 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:29.915954 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" podUID="81f49829-73ed-4760-9444-f1793861eba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:30.127590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:30.121600 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:30.127590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:30.122196 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:28:30.132668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:30.132641 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:28:30.132901 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:30.132878 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" podUID="571fc9dd-71e3-4fa7-b1af-e74f0a96654d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:30.304683 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:30.304627 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:30.325790 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:30.325677 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:30.329779 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:30.329590 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:30.339905 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:30.339022 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:28:30.352593 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:30.352541 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:28:30.817542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:30.815670 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:28:30.959705 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:30.959647 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:31.123756 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:31.123647 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:28:31.124418 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:31.124386 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:31.317658 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:31.317607 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:31.337626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:31.337580 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:28:31.359534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:31.341581 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:31.360878 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:31.360000 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:31.819208 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:31.819163 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:28:32.041174 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:32.041130 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:28:32.125878 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:32.125749 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:32.128388 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:32.128338 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:28:32.393411 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:32.363678 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:28:32.393411 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:32.366284 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:28:32.393411 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:32.366734 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:28:32.393411 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:32.380601 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:32.528541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:32.527200 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:28:32.661598 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:32.661022 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:28:32.760073 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:32.759942 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:28:32.827283 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:32.825397 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:32.885283 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:32.885155 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:33.061715 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:33.061671 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:33.157634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:33.157584 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:28:33.192460 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:33.191325 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:28:33.365885 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:33.365792 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:33.367106 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:33.367076 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:28:33.513202 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:33.513174 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:28:33.581884 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:33.581856 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:28:33.588165 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:33.588142 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:28:33.621037 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:33.620956 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:28:34.550059 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:34.550000 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:36.392733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:36.392697 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:38.151054 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:38.151019 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:40.377528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:40.371132 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:41.347311 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.347277 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj"] Oct 30 18:28:41.350611 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.350593 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.383813 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.383791 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj"] Oct 30 18:28:41.500005 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.499972 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.500175 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.500036 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.500175 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.500109 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.500254 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.500174 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.500340 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.500317 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a065f551-ec87-4081-bc18-27d0dd4234e3-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.500383 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.500363 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.500452 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.500436 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.500528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.500485 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.500599 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.500585 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvjjs\" (UniqueName: \"kubernetes.io/projected/a065f551-ec87-4081-bc18-27d0dd4234e3-kube-api-access-tvjjs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.601760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.601676 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a065f551-ec87-4081-bc18-27d0dd4234e3-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.601760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.601722 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.601760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.601745 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.602007 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.601775 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.602007 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.601820 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvjjs\" (UniqueName: \"kubernetes.io/projected/a065f551-ec87-4081-bc18-27d0dd4234e3-kube-api-access-tvjjs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.602007 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.601858 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.602188 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.602060 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.602188 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.602120 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.602188 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.602143 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.602344 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.602215 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.602344 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.602245 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.602344 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.602257 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.602489 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.602424 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.602489 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.602482 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a065f551-ec87-4081-bc18-27d0dd4234e3-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.604708 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.604677 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.604848 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.604746 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.624949 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.624927 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvjjs\" (UniqueName: \"kubernetes.io/projected/a065f551-ec87-4081-bc18-27d0dd4234e3-kube-api-access-tvjjs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.625084 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.624968 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.663313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.663290 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:41.988362 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.988336 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj"] Oct 30 18:28:41.989742 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:28:41.989716 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda065f551_ec87_4081_bc18_27d0dd4234e3.slice/crio-748a4d33053fd8e61c20b08d8f41fe2548bdb8a7ab0d6f24f3d8dc9f6ec5ac0a WatchSource:0}: Error finding container 748a4d33053fd8e61c20b08d8f41fe2548bdb8a7ab0d6f24f3d8dc9f6ec5ac0a: Status 404 returned error can't find the container with id 748a4d33053fd8e61c20b08d8f41fe2548bdb8a7ab0d6f24f3d8dc9f6ec5ac0a Oct 30 18:28:41.992692 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.992620 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:28:41.992762 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.992744 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:28:41.992807 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:41.992792 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:28:42.053660 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:42.053547 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:42.323926 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:42.323880 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" event={"ID":"a065f551-ec87-4081-bc18-27d0dd4234e3","Type":"ContainerStarted","Data":"50eaa51974dd5b8bb5f841a01c69f75f6e19d94f457d45c2ce928e68b17fb26e"} Oct 30 18:28:42.324103 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:42.323997 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" event={"ID":"a065f551-ec87-4081-bc18-27d0dd4234e3","Type":"ContainerStarted","Data":"748a4d33053fd8e61c20b08d8f41fe2548bdb8a7ab0d6f24f3d8dc9f6ec5ac0a"} Oct 30 18:28:42.371036 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:42.370986 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" podStartSLOduration=1.37097088 podStartE2EDuration="1.37097088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:42.367292845 +0000 UTC m=+2183.082862746" watchObservedRunningTime="2026-10-30 18:28:42.37097088 +0000 UTC m=+2183.086540755" Oct 30 18:28:42.663712 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:42.663626 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:43.666047 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:43.666009 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 18:28:43.666447 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:43.666094 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" podUID="a065f551-ec87-4081-bc18-27d0dd4234e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 18:28:43.740270 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:43.740236 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:44.663697 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:44.663666 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 18:28:44.663917 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:44.663722 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" podUID="a065f551-ec87-4081-bc18-27d0dd4234e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 18:28:45.541814 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:45.541776 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:45.664009 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:45.663982 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 18:28:45.664194 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:45.664030 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" podUID="a065f551-ec87-4081-bc18-27d0dd4234e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 18:28:46.664013 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:46.663976 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 18:28:46.664402 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:46.664032 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" podUID="a065f551-ec87-4081-bc18-27d0dd4234e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 18:28:47.556057 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:47.556013 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:47.667530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:47.665566 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 18:28:47.667530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:47.665643 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" podUID="a065f551-ec87-4081-bc18-27d0dd4234e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 18:28:48.214592 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.213034 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:48.602346 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.602315 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v"] Oct 30 18:28:48.605439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.605416 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.621834 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.620075 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-ac632e-openshift-default-dockercfg-vbqgb\"" Oct 30 18:28:48.641037 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.641009 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v"] Oct 30 18:28:48.692933 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.692899 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:48.755461 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.755428 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-data\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.755833 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.755809 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-token\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.756036 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.756018 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fh94\" (UniqueName: \"kubernetes.io/projected/67142e99-ffa0-4b21-8f93-735d5663c87e-kube-api-access-2fh94\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.756251 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.756236 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-credential-socket\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.756447 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.756431 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-workload-certs\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.756659 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.756642 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67142e99-ffa0-4b21-8f93-735d5663c87e-istiod-ca-cert\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.756800 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.756786 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-workload-socket\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.757054 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.757035 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-podinfo\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.794855 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.794822 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-envoy\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.898079 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.897935 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-podinfo\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.898988 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.898962 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-envoy\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.899686 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.899666 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-data\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.900177 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.900158 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-token\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.907280 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.907251 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fh94\" (UniqueName: \"kubernetes.io/projected/67142e99-ffa0-4b21-8f93-735d5663c87e-kube-api-access-2fh94\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.908150 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.900064 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-data\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.908150 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.906245 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:48.916622 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.909959 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-credential-socket\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.917125 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.917055 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-envoy\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.917296 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.917277 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-credential-socket\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.917606 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.917579 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-workload-certs\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.927674 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.921575 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-podinfo\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.927934 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.927447 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-workload-certs\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.929081 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.928973 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67142e99-ffa0-4b21-8f93-735d5663c87e-istiod-ca-cert\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.929929 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.929801 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67142e99-ffa0-4b21-8f93-735d5663c87e-istiod-ca-cert\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.930174 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.930156 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-workload-socket\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.930852 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.930832 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-workload-socket\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.945526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.942197 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-token\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:48.973397 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:48.971196 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fh94\" (UniqueName: \"kubernetes.io/projected/67142e99-ffa0-4b21-8f93-735d5663c87e-kube-api-access-2fh94\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:49.223266 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:49.223174 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:49.242964 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:49.242940 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:49.243850 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:49.243825 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:49.245775 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:49.245754 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:28:49.344183 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:49.344150 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:49.584141 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:49.584108 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v"] Oct 30 18:28:49.585490 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:28:49.585464 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67142e99_ffa0_4b21_8f93_735d5663c87e.slice/crio-eb8a220d7af35432580047a45b424e24f681a2732104ac748f66cc209070ae5b WatchSource:0}: Error finding container eb8a220d7af35432580047a45b424e24f681a2732104ac748f66cc209070ae5b: Status 404 returned error can't find the container with id eb8a220d7af35432580047a45b424e24f681a2732104ac748f66cc209070ae5b Oct 30 18:28:49.588126 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:49.588089 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:28:49.588237 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:49.588171 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:28:49.588237 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:49.588216 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:28:49.685291 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:49.685204 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:50.539255 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:50.539215 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" event={"ID":"67142e99-ffa0-4b21-8f93-735d5663c87e","Type":"ContainerStarted","Data":"8aa33ec0ada9b6affb208856da6a1cfbbfa39a7f97414d9d8cc44e1bd2b210f6"} Oct 30 18:28:50.539692 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:50.539305 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" event={"ID":"67142e99-ffa0-4b21-8f93-735d5663c87e","Type":"ContainerStarted","Data":"eb8a220d7af35432580047a45b424e24f681a2732104ac748f66cc209070ae5b"} Oct 30 18:28:50.596883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:50.596826 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" podStartSLOduration=2.596808715 podStartE2EDuration="2.596808715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:50.594718033 +0000 UTC m=+2191.310287908" watchObservedRunningTime="2026-10-30 18:28:50.596808715 +0000 UTC m=+2191.312378590" Oct 30 18:28:51.085164 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:51.085133 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:51.225659 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:51.223494 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:51.231418 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:51.231391 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:51.545931 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:51.545431 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:51.546458 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:51.546297 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:28:51.929702 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:28:51.929627 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:28:53.097376 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:53.097338 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:53.270524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:53.268375 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:54.783808 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:54.783772 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:54.926848 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:54.926811 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:55.486955 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:55.486913 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:57.161389 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:57.161357 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:58.820602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:28:58.820567 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:00.476460 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:00.476416 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:02.578604 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:02.574121 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:04.138176 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:04.138143 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:04.668536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:04.668484 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:05.064242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.064213 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv"] Oct 30 18:29:05.068547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.068494 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.084844 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.084810 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-a2a10a-openshift-default-dockercfg-4fw27\"" Oct 30 18:29:05.102318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.102291 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv"] Oct 30 18:29:05.153522 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.153478 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:05.257030 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.256993 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h4v4\" (UniqueName: \"kubernetes.io/projected/a2d95ad6-c697-48a0-8592-0ff95ce3767b-kube-api-access-9h4v4\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.257392 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.257368 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-token\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.257774 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.257752 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istiod-ca-cert\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.257966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.257950 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-workload-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.269528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.258137 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-envoy\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.270631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.270611 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-workload-certs\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.270846 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.270826 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-data\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.271034 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.271017 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-credential-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.271205 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.271190 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-podinfo\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.310472 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.310439 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:05.374553 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.374434 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-data\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.374829 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.374805 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-credential-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.374956 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.374940 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-podinfo\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.375226 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.375208 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h4v4\" (UniqueName: \"kubernetes.io/projected/a2d95ad6-c697-48a0-8592-0ff95ce3767b-kube-api-access-9h4v4\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.375399 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.375376 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-token\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.375710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.375694 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istiod-ca-cert\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.375843 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.375828 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-workload-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.375979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.375965 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-envoy\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.376103 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.376089 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-workload-certs\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.376697 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.376675 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-workload-certs\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.377099 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.377079 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-data\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.388618 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.377776 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-credential-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.388798 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.378413 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-workload-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.388937 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.379033 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istiod-ca-cert\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.389719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.389697 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-podinfo\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.422130 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.422099 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-token\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.435623 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.435576 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h4v4\" (UniqueName: \"kubernetes.io/projected/a2d95ad6-c697-48a0-8592-0ff95ce3767b-kube-api-access-9h4v4\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.441972 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.441938 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-envoy\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.688661 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.688582 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:05.845040 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:05.845002 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:06.023339 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:06.023303 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv"] Oct 30 18:29:06.024308 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:29:06.024277 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d95ad6_c697_48a0_8592_0ff95ce3767b.slice/crio-a0a112f2f6b1cd937582f84d851d5724311e1e73c92c8d5d9a36dcf22db76d68 WatchSource:0}: Error finding container a0a112f2f6b1cd937582f84d851d5724311e1e73c92c8d5d9a36dcf22db76d68: Status 404 returned error can't find the container with id a0a112f2f6b1cd937582f84d851d5724311e1e73c92c8d5d9a36dcf22db76d68 Oct 30 18:29:06.027100 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:06.027066 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:06.027165 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:06.027140 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:06.027202 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:06.027168 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:06.703379 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:06.703345 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" event={"ID":"a2d95ad6-c697-48a0-8592-0ff95ce3767b","Type":"ContainerStarted","Data":"69c0db244afc76ed20a51a416413b30baf9462d999fb95e2b0b5f000b1f60a21"} Oct 30 18:29:06.703789 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:06.703400 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" event={"ID":"a2d95ad6-c697-48a0-8592-0ff95ce3767b","Type":"ContainerStarted","Data":"a0a112f2f6b1cd937582f84d851d5724311e1e73c92c8d5d9a36dcf22db76d68"} Oct 30 18:29:06.773886 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:06.773841 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" podStartSLOduration=1.773828479 podStartE2EDuration="1.773828479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:06.758333692 +0000 UTC m=+2207.473903577" watchObservedRunningTime="2026-10-30 18:29:06.773828479 +0000 UTC m=+2207.489398353" Oct 30 18:29:07.507102 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:07.507057 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:07.664529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:07.662028 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:07.696077 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:07.692035 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:07.725580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:07.720844 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:07.725580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:07.724522 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:07.726239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:07.725969 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:29:07.810388 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:07.810353 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:09.191069 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:09.191029 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:09.383919 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:09.383867 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:09.551470 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:09.551341 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:11.007702 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:11.007666 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:11.166173 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:11.166139 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:11.973396 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:11.973359 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:13.769527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:13.761923 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:15.467985 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:15.467946 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:17.147460 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:17.147427 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:17.296418 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:17.296389 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:17.862411 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:17.862341 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:18.070941 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.070910 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw"] Oct 30 18:29:18.074192 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.074172 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.090908 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.090882 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-e77b1d-openshift-default-dockercfg-k7wxj\"" Oct 30 18:29:18.108285 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.108255 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw"] Oct 30 18:29:18.159968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.159374 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-token\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.159968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.159459 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-workload-certs\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.159968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.159518 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqvxs\" (UniqueName: \"kubernetes.io/projected/0baa3ccf-73e5-4555-9e51-5c627b74124d-kube-api-access-jqvxs\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.159968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.159606 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0baa3ccf-73e5-4555-9e51-5c627b74124d-istiod-ca-cert\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.159968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.159699 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-envoy\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.159968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.159812 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-credential-socket\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.159968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.159893 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-workload-socket\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.159968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.159938 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-podinfo\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.159968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.159959 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-data\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.163600 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.163576 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:18.261072 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.261042 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-podinfo\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.261207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.261077 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-data\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.261207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.261141 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-token\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.261338 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.261314 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-workload-certs\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.261388 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.261360 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvxs\" (UniqueName: \"kubernetes.io/projected/0baa3ccf-73e5-4555-9e51-5c627b74124d-kube-api-access-jqvxs\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.261447 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.261422 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0baa3ccf-73e5-4555-9e51-5c627b74124d-istiod-ca-cert\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.261527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.261465 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-data\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.261527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.261487 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-envoy\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.261656 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.261566 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-credential-socket\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.261719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.261678 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-workload-socket\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.261765 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.261711 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-workload-certs\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.261957 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.261933 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-credential-socket\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.262066 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.261974 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-workload-socket\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.262291 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.262115 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0baa3ccf-73e5-4555-9e51-5c627b74124d-istiod-ca-cert\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.263794 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.263773 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-envoy\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.263893 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.263811 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-podinfo\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.289432 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.289411 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-token\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.289969 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.289946 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqvxs\" (UniqueName: \"kubernetes.io/projected/0baa3ccf-73e5-4555-9e51-5c627b74124d-kube-api-access-jqvxs\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.375747 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.375722 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np"] Oct 30 18:29:18.379621 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.379602 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.385145 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.385120 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:18.391716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.391697 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:18.396645 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.396256 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-50e9fc-openshift-default-dockercfg-rpx9w\"" Oct 30 18:29:18.421702 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.421674 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np"] Oct 30 18:29:18.465051 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.465011 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dba942a9-7d94-415a-8491-a17212be1601-istiod-ca-cert\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.465171 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.465136 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-workload-certs\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.465330 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.465296 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-workload-socket\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.465399 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.465348 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-istio-envoy\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.465399 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.465393 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dba942a9-7d94-415a-8491-a17212be1601-istio-podinfo\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.465538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.465479 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-istio-data\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.465603 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.465581 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dba942a9-7d94-415a-8491-a17212be1601-istio-token\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.465691 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.465623 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-credential-socket\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.465691 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.465673 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrs9n\" (UniqueName: \"kubernetes.io/projected/dba942a9-7d94-415a-8491-a17212be1601-kube-api-access-nrs9n\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.578529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.577590 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-workload-socket\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.578529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.577642 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-istio-envoy\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.578529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.577688 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dba942a9-7d94-415a-8491-a17212be1601-istio-podinfo\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.578529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.577737 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-istio-data\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.578529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.577801 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dba942a9-7d94-415a-8491-a17212be1601-istio-token\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.578529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.577843 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-credential-socket\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.578529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.577894 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrs9n\" (UniqueName: \"kubernetes.io/projected/dba942a9-7d94-415a-8491-a17212be1601-kube-api-access-nrs9n\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.578529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.578170 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dba942a9-7d94-415a-8491-a17212be1601-istiod-ca-cert\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.578529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.578230 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-workload-certs\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.578529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.578257 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-istio-data\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.631531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.611640 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dba942a9-7d94-415a-8491-a17212be1601-istiod-ca-cert\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.631531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.611905 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-workload-certs\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.631531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.615706 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-workload-socket\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.631531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.627223 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-credential-socket\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.687588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.665226 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:18.895019 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.889902 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-istio-envoy\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.895019 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.893042 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dba942a9-7d94-415a-8491-a17212be1601-istio-token\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.910020 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.903050 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:18.919692 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.919607 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dba942a9-7d94-415a-8491-a17212be1601-istio-podinfo\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.939737 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.939621 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrs9n\" (UniqueName: \"kubernetes.io/projected/dba942a9-7d94-415a-8491-a17212be1601-kube-api-access-nrs9n\") pod \"e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:18.971232 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.971189 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw"] Oct 30 18:29:18.985660 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.978218 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:18.985660 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.978313 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:18.985660 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:18.978360 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:19.018553 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:19.018479 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:19.353340 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:19.353312 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np"] Oct 30 18:29:19.354878 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:29:19.354851 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba942a9_7d94_415a_8491_a17212be1601.slice/crio-fded3bb65a2a83c53ec2a723501073b82a627b4836c5dbc954f0974a1e973df7 WatchSource:0}: Error finding container fded3bb65a2a83c53ec2a723501073b82a627b4836c5dbc954f0974a1e973df7: Status 404 returned error can't find the container with id fded3bb65a2a83c53ec2a723501073b82a627b4836c5dbc954f0974a1e973df7 Oct 30 18:29:19.358160 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:19.358127 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:19.358255 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:19.358192 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:19.358255 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:19.358221 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:19.689329 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:19.689224 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:19.961924 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:19.961842 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" event={"ID":"0baa3ccf-73e5-4555-9e51-5c627b74124d","Type":"ContainerStarted","Data":"e627332c30048dc62c7abd519e19bafc165413f069b36427fb4456800e4971e9"} Oct 30 18:29:19.962097 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:19.961937 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" event={"ID":"0baa3ccf-73e5-4555-9e51-5c627b74124d","Type":"ContainerStarted","Data":"574511a27d2bf96a841fd2b1584a55dfcc90fed775d02f215f6e54093eb987a4"} Oct 30 18:29:19.963690 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:19.963663 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" event={"ID":"dba942a9-7d94-415a-8491-a17212be1601","Type":"ContainerStarted","Data":"38a01e2bc6fb4ba12131c55003733c74e3fff4f17fb808240979c542a9be32b4"} Oct 30 18:29:19.963812 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:19.963713 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" event={"ID":"dba942a9-7d94-415a-8491-a17212be1601","Type":"ContainerStarted","Data":"fded3bb65a2a83c53ec2a723501073b82a627b4836c5dbc954f0974a1e973df7"} Oct 30 18:29:20.019179 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:20.019155 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:20.024253 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:20.024235 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:20.034271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:20.034231 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" podStartSLOduration=2.034219469 podStartE2EDuration="2.034219469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:20.020993679 +0000 UTC m=+2220.736563552" watchObservedRunningTime="2026-10-30 18:29:20.034219469 +0000 UTC m=+2220.749789406" Oct 30 18:29:20.068060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:20.068021 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" podStartSLOduration=2.068008482 podStartE2EDuration="2.068008482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:20.065274212 +0000 UTC m=+2220.780844086" watchObservedRunningTime="2026-10-30 18:29:20.068008482 +0000 UTC m=+2220.783578356" Oct 30 18:29:20.392543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:20.392494 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:20.398155 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:20.398129 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:20.970691 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:20.970659 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:20.970933 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:20.970730 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:20.971522 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:20.971480 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:29:20.971645 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:20.971580 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:29:21.456112 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:21.456073 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:21.930557 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:29:21.930487 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:29:22.483683 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:22.483654 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:22.639980 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:22.639942 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:22.823019 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:22.822984 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:23.505339 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:23.505303 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:24.227253 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:24.227218 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:24.390960 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:24.390927 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:25.448787 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:25.448749 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:26.302002 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:26.301966 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:26.466943 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:26.466906 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:27.258727 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:27.258692 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:27.421378 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:27.421335 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:28.537528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:28.535324 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:29.004286 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:29.004196 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:29.263453 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:29.263360 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:29.846982 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:29.846946 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:30.132583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:30.131790 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:31.684570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:31.681894 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:33.354357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:33.354318 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:33.517704 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:33.517677 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:35.098405 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:35.098374 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:35.244774 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:35.244738 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:36.807134 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:36.807094 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:39.211987 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:39.211949 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:40.820091 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:40.820054 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:40.996529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:40.994102 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:41.150243 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:41.150164 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:42.513189 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:42.513148 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:42.677144 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:42.677110 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:44.268963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:44.268927 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:45.860443 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:45.860410 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:47.657274 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:47.657239 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:49.376543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:49.376486 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:51.064145 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:51.064115 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:51.933623 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:29:51.933587 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:29:53.308070 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:53.308027 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:54.932279 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:54.932240 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:55.091474 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:55.091431 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:55.346243 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:55.346211 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:55.503161 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:55.503128 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:55.890041 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:55.890006 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf"] Oct 30 18:29:55.895546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:55.895520 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:55.928677 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:55.928645 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-b00d69-openshift-default-dockercfg-dp9gr\"" Oct 30 18:29:55.934630 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:55.934605 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf"] Oct 30 18:29:55.978593 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:55.978556 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:56.039419 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.039377 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.039742 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.039717 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.039983 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.039963 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zmvg\" (UniqueName: \"kubernetes.io/projected/0815ac39-de71-4bc3-98d3-28c88a9d2139-kube-api-access-4zmvg\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.040109 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.040093 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.040222 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.040207 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.040399 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.040382 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0815ac39-de71-4bc3-98d3-28c88a9d2139-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.040627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.040609 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.040796 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.040779 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.040944 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.040929 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.151632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.148609 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.151632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.148681 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.151632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.148803 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zmvg\" (UniqueName: \"kubernetes.io/projected/0815ac39-de71-4bc3-98d3-28c88a9d2139-kube-api-access-4zmvg\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.151632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.148838 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.151632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.148868 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.151632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.148954 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0815ac39-de71-4bc3-98d3-28c88a9d2139-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.151632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.149000 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.151632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.149056 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.151632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.149113 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.151632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.149696 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.151632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.149961 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.151632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.150220 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.153272 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.153243 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0815ac39-de71-4bc3-98d3-28c88a9d2139-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.154446 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.154389 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.168529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.163151 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.194707 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.194673 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zmvg\" (UniqueName: \"kubernetes.io/projected/0815ac39-de71-4bc3-98d3-28c88a9d2139-kube-api-access-4zmvg\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.204527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.197035 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.227545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.215479 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.227545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.215927 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:56.363043 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.363010 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:56.931712 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.931686 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf"] Oct 30 18:29:56.932660 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:29:56.932630 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0815ac39_de71_4bc3_98d3_28c88a9d2139.slice/crio-a117584a514a6056c2485be52a8a8e63088b3d5fb6389df917dd23a741b82d72 WatchSource:0}: Error finding container a117584a514a6056c2485be52a8a8e63088b3d5fb6389df917dd23a741b82d72: Status 404 returned error can't find the container with id a117584a514a6056c2485be52a8a8e63088b3d5fb6389df917dd23a741b82d72 Oct 30 18:29:56.935731 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.935695 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:56.936071 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.935767 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:56.936071 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:56.935800 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:57.052314 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:57.052276 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:57.373377 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:57.373339 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" event={"ID":"0815ac39-de71-4bc3-98d3-28c88a9d2139","Type":"ContainerStarted","Data":"7e367247a2f0e8c781fafc99df6353df88fab7c94e13841a9887e8c5c86603da"} Oct 30 18:29:57.373560 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:57.373410 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" event={"ID":"0815ac39-de71-4bc3-98d3-28c88a9d2139","Type":"ContainerStarted","Data":"a117584a514a6056c2485be52a8a8e63088b3d5fb6389df917dd23a741b82d72"} Oct 30 18:29:57.435959 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:57.435899 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" podStartSLOduration=2.435880796 podStartE2EDuration="2.435880796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:57.42965094 +0000 UTC m=+2258.145220828" watchObservedRunningTime="2026-10-30 18:29:57.435880796 +0000 UTC m=+2258.151450673" Oct 30 18:29:58.146844 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:58.146811 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:29:58.216339 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:58.216309 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:58.222125 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:58.222101 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:58.452586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:58.449155 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:58.474530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:58.472874 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:29:58.860779 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:58.860743 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:59.932105 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:29:59.932071 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:00.135719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.135674 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:00.159104 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.159070 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg"] Oct 30 18:30:00.170720 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.170691 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg" Oct 30 18:30:00.384530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.379002 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6lz8\" (UniqueName: \"kubernetes.io/projected/80914a10-e845-4294-9423-7d356e84d57b-kube-api-access-r6lz8\") pod \"maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg\" (UID: \"80914a10-e845-4294-9423-7d356e84d57b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg" Oct 30 18:30:00.420291 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.419661 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80914a10-e845-4294-9423-7d356e84d57b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg\" (UID: \"80914a10-e845-4294-9423-7d356e84d57b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg" Oct 30 18:30:00.610570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.468732 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:30:00.610570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.554072 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80914a10-e845-4294-9423-7d356e84d57b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg\" (UID: \"80914a10-e845-4294-9423-7d356e84d57b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg" Oct 30 18:30:00.610570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.554349 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6lz8\" (UniqueName: \"kubernetes.io/projected/80914a10-e845-4294-9423-7d356e84d57b-kube-api-access-r6lz8\") pod \"maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg\" (UID: \"80914a10-e845-4294-9423-7d356e84d57b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg" Oct 30 18:30:00.616846 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.616799 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj"] Oct 30 18:30:00.760680 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.760600 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80914a10-e845-4294-9423-7d356e84d57b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg\" (UID: \"80914a10-e845-4294-9423-7d356e84d57b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg" Oct 30 18:30:00.784342 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.784290 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg"] Oct 30 18:30:00.811676 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.811645 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw"] Oct 30 18:30:00.817515 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.817473 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22"] Oct 30 18:30:00.884428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.824631 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj" Oct 30 18:30:00.884428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.831602 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw" Oct 30 18:30:00.884428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.870182 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6lz8\" (UniqueName: \"kubernetes.io/projected/80914a10-e845-4294-9423-7d356e84d57b-kube-api-access-r6lz8\") pod \"maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg\" (UID: \"80914a10-e845-4294-9423-7d356e84d57b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg" Oct 30 18:30:00.899525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.887674 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q"] Oct 30 18:30:00.920684 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.892746 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22" Oct 30 18:30:00.935547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.934362 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q" Oct 30 18:30:00.998326 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.998249 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5184124-242d-445e-9341-59d13045fda9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889750-v4r4q\" (UID: \"c5184124-242d-445e-9341-59d13045fda9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q" Oct 30 18:30:00.998619 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.998591 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsjcw\" (UniqueName: \"kubernetes.io/projected/c5184124-242d-445e-9341-59d13045fda9-kube-api-access-nsjcw\") pod \"maas-api-key-cleanup-29889750-v4r4q\" (UID: \"c5184124-242d-445e-9341-59d13045fda9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q" Oct 30 18:30:00.998821 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.998801 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf18bbde-1895-48b6-9ad1-7f6560660f8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22\" (UID: \"cf18bbde-1895-48b6-9ad1-7f6560660f8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22" Oct 30 18:30:00.999087 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.999066 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be072889-51a1-4341-8559-6f445961c745-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw\" (UID: \"be072889-51a1-4341-8559-6f445961c745\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw" Oct 30 18:30:00.999289 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.999270 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9efc340c-4b4f-4a7f-ace1-0800c843361a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj\" (UID: \"9efc340c-4b4f-4a7f-ace1-0800c843361a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj" Oct 30 18:30:00.999553 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.999529 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np4p5\" (UniqueName: \"kubernetes.io/projected/9efc340c-4b4f-4a7f-ace1-0800c843361a-kube-api-access-np4p5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj\" (UID: \"9efc340c-4b4f-4a7f-ace1-0800c843361a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj" Oct 30 18:30:00.999774 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.999753 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scs8l\" (UniqueName: \"kubernetes.io/projected/be072889-51a1-4341-8559-6f445961c745-kube-api-access-scs8l\") pod \"maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw\" (UID: \"be072889-51a1-4341-8559-6f445961c745\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw" Oct 30 18:30:00.999936 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:00.999918 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dff87\" (UniqueName: \"kubernetes.io/projected/cf18bbde-1895-48b6-9ad1-7f6560660f8b-kube-api-access-dff87\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22\" (UID: \"cf18bbde-1895-48b6-9ad1-7f6560660f8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22" Oct 30 18:30:01.007321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.007297 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg" Oct 30 18:30:01.032606 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.032558 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw"] Oct 30 18:30:01.082596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.080231 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj"] Oct 30 18:30:01.113657 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.113621 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf18bbde-1895-48b6-9ad1-7f6560660f8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22\" (UID: \"cf18bbde-1895-48b6-9ad1-7f6560660f8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22" Oct 30 18:30:01.114063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.114008 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be072889-51a1-4341-8559-6f445961c745-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw\" (UID: \"be072889-51a1-4341-8559-6f445961c745\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw" Oct 30 18:30:01.114292 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.114271 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9efc340c-4b4f-4a7f-ace1-0800c843361a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj\" (UID: \"9efc340c-4b4f-4a7f-ace1-0800c843361a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj" Oct 30 18:30:01.114469 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.114450 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np4p5\" (UniqueName: \"kubernetes.io/projected/9efc340c-4b4f-4a7f-ace1-0800c843361a-kube-api-access-np4p5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj\" (UID: \"9efc340c-4b4f-4a7f-ace1-0800c843361a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj" Oct 30 18:30:01.114655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.114638 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scs8l\" (UniqueName: \"kubernetes.io/projected/be072889-51a1-4341-8559-6f445961c745-kube-api-access-scs8l\") pod \"maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw\" (UID: \"be072889-51a1-4341-8559-6f445961c745\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw" Oct 30 18:30:01.114783 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.114769 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dff87\" (UniqueName: \"kubernetes.io/projected/cf18bbde-1895-48b6-9ad1-7f6560660f8b-kube-api-access-dff87\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22\" (UID: \"cf18bbde-1895-48b6-9ad1-7f6560660f8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22" Oct 30 18:30:01.114925 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.114912 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5184124-242d-445e-9341-59d13045fda9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889750-v4r4q\" (UID: \"c5184124-242d-445e-9341-59d13045fda9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q" Oct 30 18:30:01.115028 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.115016 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsjcw\" (UniqueName: \"kubernetes.io/projected/c5184124-242d-445e-9341-59d13045fda9-kube-api-access-nsjcw\") pod \"maas-api-key-cleanup-29889750-v4r4q\" (UID: \"c5184124-242d-445e-9341-59d13045fda9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q" Oct 30 18:30:01.117219 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.117193 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf18bbde-1895-48b6-9ad1-7f6560660f8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22\" (UID: \"cf18bbde-1895-48b6-9ad1-7f6560660f8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22" Oct 30 18:30:01.118182 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.118159 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be072889-51a1-4341-8559-6f445961c745-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw\" (UID: \"be072889-51a1-4341-8559-6f445961c745\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw" Oct 30 18:30:01.126746 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.126724 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5184124-242d-445e-9341-59d13045fda9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889750-v4r4q\" (UID: \"c5184124-242d-445e-9341-59d13045fda9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q" Oct 30 18:30:01.126979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.126959 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22"] Oct 30 18:30:01.127709 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.127684 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9efc340c-4b4f-4a7f-ace1-0800c843361a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj\" (UID: \"9efc340c-4b4f-4a7f-ace1-0800c843361a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj" Oct 30 18:30:01.256841 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.255401 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsjcw\" (UniqueName: \"kubernetes.io/projected/c5184124-242d-445e-9341-59d13045fda9-kube-api-access-nsjcw\") pod \"maas-api-key-cleanup-29889750-v4r4q\" (UID: \"c5184124-242d-445e-9341-59d13045fda9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q" Oct 30 18:30:01.320541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.304269 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q" Oct 30 18:30:01.331923 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.314721 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:01.331923 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.314760 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:01.364534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.352362 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np4p5\" (UniqueName: \"kubernetes.io/projected/9efc340c-4b4f-4a7f-ace1-0800c843361a-kube-api-access-np4p5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj\" (UID: \"9efc340c-4b4f-4a7f-ace1-0800c843361a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj" Oct 30 18:30:01.402480 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.396257 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scs8l\" (UniqueName: \"kubernetes.io/projected/be072889-51a1-4341-8559-6f445961c745-kube-api-access-scs8l\") pod \"maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw\" (UID: \"be072889-51a1-4341-8559-6f445961c745\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw" Oct 30 18:30:01.425734 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.423239 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dff87\" (UniqueName: \"kubernetes.io/projected/cf18bbde-1895-48b6-9ad1-7f6560660f8b-kube-api-access-dff87\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22\" (UID: \"cf18bbde-1895-48b6-9ad1-7f6560660f8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22" Oct 30 18:30:01.426759 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.426732 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22" Oct 30 18:30:01.529555 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.520288 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw" Oct 30 18:30:01.842801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.842751 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:30:01.937563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:01.935677 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:30:02.297969 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:02.216638 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:02.358703 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:02.336626 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:02.558198 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:02.467775 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:02.812146 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:02.812101 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj" Oct 30 18:30:03.003529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:02.982060 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:03.003529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:02.983652 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:30:03.110593 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:03.050584 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:30:03.347777 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:03.108543 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:03.350402 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:03.293219 4603 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.89s" Oct 30 18:30:03.395603 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:03.377423 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:03.462351 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:03.462306 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:30:03.622144 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:03.321217 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:30:03.644753 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:03.644688 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:30:03.650586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:03.648630 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:03.650586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:03.650303 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:30:03.677725 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:03.677694 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:30:03.815739 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:03.814461 4603 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-wjstn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:30:03.815739 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:03.814544 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wjstn" podUID="c77c24c9-98c1-43c8-9460-e5b3c509c6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:04.075617 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:04.075565 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:04.287596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:04.281936 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:30:04.287596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:04.281999 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" podUID="187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:04.328039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:04.327603 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:04.391429 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:04.335407 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:04.391429 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:04.365150 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:04.452085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:04.451953 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:30:04.452085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:04.452020 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" podUID="b6e43478-8cc1-4452-8bc8-fed14f127896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:04.475865 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:04.475831 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:30:04.485448 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:04.485416 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q"] Oct 30 18:30:04.568982 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:04.550919 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:30:04.568982 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:04.550967 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc"] Oct 30 18:30:04.700129 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:04.675815 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:04.700129 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:04.689439 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:30:04.704634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:04.702596 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:30:04.704634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:04.703318 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" podUID="f4343775-794f-4a3c-8732-01eeff138bbe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Oct 30 18:30:04.777732 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:04.777665 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:30:04.936053 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:04.819080 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:05.007535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:04.848291 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:05.175539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:05.147895 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:30:05.357024 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:05.356974 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:05.380552 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:05.365720 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:30:05.551818 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:05.432732 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc"] Oct 30 18:30:05.594687 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:05.594650 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg"] Oct 30 18:30:05.595111 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:05.595088 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:30:05.708662 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:05.432839 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc" Oct 30 18:30:05.745263 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:05.522267 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Oct 30 18:30:05.819753 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:05.802609 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:05.819753 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:05.588790 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hgp4\" (UniqueName: \"kubernetes.io/projected/103f4661-1a62-4f4d-8089-3ceabd3271ad-kube-api-access-9hgp4\") pod \"payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc\" (UID: \"103f4661-1a62-4f4d-8089-3ceabd3271ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc" Oct 30 18:30:05.819753 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:05.804319 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103f4661-1a62-4f4d-8089-3ceabd3271ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc\" (UID: \"103f4661-1a62-4f4d-8089-3ceabd3271ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc" Oct 30 18:30:05.819753 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:05.804708 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:30:05.947585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:05.869907 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-b00d69\"" Oct 30 18:30:06.008584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:05.968933 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:30:06.008584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:05.969349 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:30:06.072585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:06.069423 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:30:06.172258 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:06.152994 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103f4661-1a62-4f4d-8089-3ceabd3271ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc\" (UID: \"103f4661-1a62-4f4d-8089-3ceabd3271ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc" Oct 30 18:30:06.205536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:06.204881 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hgp4\" (UniqueName: \"kubernetes.io/projected/103f4661-1a62-4f4d-8089-3ceabd3271ad-kube-api-access-9hgp4\") pod \"payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc\" (UID: \"103f4661-1a62-4f4d-8089-3ceabd3271ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc" Oct 30 18:30:06.205536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:06.171751 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-b00d69-dockercfg-zgpd7\"" Oct 30 18:30:06.207614 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:06.207564 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:06.695725 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:06.636181 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:06.837572 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:06.820309 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103f4661-1a62-4f4d-8089-3ceabd3271ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc\" (UID: \"103f4661-1a62-4f4d-8089-3ceabd3271ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc" Oct 30 18:30:06.860529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:06.857205 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hgp4\" (UniqueName: \"kubernetes.io/projected/103f4661-1a62-4f4d-8089-3ceabd3271ad-kube-api-access-9hgp4\") pod \"payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc\" (UID: \"103f4661-1a62-4f4d-8089-3ceabd3271ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc" Oct 30 18:30:06.897529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:06.781493 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:30:06.915555 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:06.900539 4603 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.305s" Oct 30 18:30:06.915840 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:06.915813 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf"] Oct 30 18:30:06.988094 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:06.951568 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc" Oct 30 18:30:07.004760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:06.999822 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:30:07.110532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:07.085087 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:30:07.110532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:07.087312 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:07.161288 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:07.121692 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Oct 30 18:30:07.247583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:07.125783 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:07.248602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:07.126893 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:30:07.692238 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:07.190749 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:07.808576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:07.227624 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:30:07.842879 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:07.842812 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:30:07.889541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:07.861454 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded" Oct 30 18:30:08.211101 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:08.050642 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:30:08.264717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:08.050790 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf" Oct 30 18:30:08.533597 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:08.519990 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:30:08.533597 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:08.527654 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:08.535463 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:08.099627 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:08.547713 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:08.099664 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:08.547993 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:08.190151 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:08.672529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:08.670245 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 30 18:30:08.727538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:08.727125 4603 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.022487842s: [/var/lib/containers/storage/overlay/88a1a980b03ed3b477123ef050685ebc55d973880248e157072bc2d23fa75701/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 18:30:08.781529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:08.750594 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:30:08.781529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:08.750677 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" podUID="58e2469b-dc0d-40ab-bd74-a47c4a9e438b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:08.781529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:08.771777 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:08.863581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:08.830022 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9b49ca-d057-4d90-bab8-1519401aa457-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf\" (UID: \"cb9b49ca-d057-4d90-bab8-1519401aa457\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf" Oct 30 18:30:09.022528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:09.007623 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcl2j\" (UniqueName: \"kubernetes.io/projected/cb9b49ca-d057-4d90-bab8-1519401aa457-kube-api-access-rcl2j\") pod \"payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf\" (UID: \"cb9b49ca-d057-4d90-bab8-1519401aa457\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf" Oct 30 18:30:09.026528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:08.911841 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:09.034528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:08.911913 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:09.054536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:08.948609 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:30:09.212676 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:08.890230 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:30:09.212676 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:09.208876 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" podUID="f7882513-ef6d-4b11-871e-39a29ad022df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:09.239356 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:09.239301 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:30:09.484584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:09.464128 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:09.533531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:09.533287 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:09.533798 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:09.533765 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:30:09.590159 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:09.569206 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:09.590159 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:09.570008 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:09.634529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:09.597115 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcl2j\" (UniqueName: \"kubernetes.io/projected/cb9b49ca-d057-4d90-bab8-1519401aa457-kube-api-access-rcl2j\") pod \"payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf\" (UID: \"cb9b49ca-d057-4d90-bab8-1519401aa457\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf" Oct 30 18:30:09.634529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:09.597280 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9b49ca-d057-4d90-bab8-1519401aa457-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf\" (UID: \"cb9b49ca-d057-4d90-bab8-1519401aa457\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf" Oct 30 18:30:09.766770 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:09.748397 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9b49ca-d057-4d90-bab8-1519401aa457-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf\" (UID: \"cb9b49ca-d057-4d90-bab8-1519401aa457\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf" Oct 30 18:30:10.050643 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:10.050593 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:10.051381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:10.051350 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:30:10.224624 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:10.217090 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:30:10.224624 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:10.217146 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" podUID="e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:10.292980 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:10.292586 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:10.309673 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:10.309587 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:10.627647 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:10.365678 4603 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.519s" Oct 30 18:30:10.627647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:10.612890 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg" event={"ID":"80914a10-e845-4294-9423-7d356e84d57b","Type":"ContainerStarted","Data":"48960a248f2439b5b385f302de2050fe7e3fdcf862c2e3fe27be829b3a794962"} Oct 30 18:30:10.627647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:10.406094 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcl2j\" (UniqueName: \"kubernetes.io/projected/cb9b49ca-d057-4d90-bab8-1519401aa457-kube-api-access-rcl2j\") pod \"payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf\" (UID: \"cb9b49ca-d057-4d90-bab8-1519401aa457\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf" Oct 30 18:30:10.627647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:10.533529 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:10.635526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:10.632645 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:10.635685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:10.534661 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:10.698650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:10.611617 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:10.698886 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:10.554714 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:10.816571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:10.462266 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:10.876340 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:10.851295 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf" Oct 30 18:30:11.098665 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:11.098600 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:11.099738 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:11.099686 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:30:11.184668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:11.153548 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf"] Oct 30 18:30:11.184668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:11.153684 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:30:11.184668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:11.153728 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:30:11.184668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:11.153764 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:30:11.184668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:11.153780 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw" event={"ID":"be072889-51a1-4341-8559-6f445961c745","Type":"ContainerStarted","Data":"c1c987a99b7154bc5348eecf54b046e08a97c05d8130c6f8c0970fc2195a59c5"} Oct 30 18:30:11.184668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:11.153812 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22" event={"ID":"cf18bbde-1895-48b6-9ad1-7f6560660f8b","Type":"ContainerStarted","Data":"3eaf191ab371afbb4e52edafc000411222d0d023e0017dde3d9f7c6eb4ff0952"} Oct 30 18:30:11.184668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:11.153850 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22" event={"ID":"cf18bbde-1895-48b6-9ad1-7f6560660f8b","Type":"ContainerStarted","Data":"4206f866113caa22bf41ea465179a81ba349e15ea906c129ee2be1529aed56d3"} Oct 30 18:30:11.184668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:11.153871 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q" event={"ID":"c5184124-242d-445e-9341-59d13045fda9","Type":"ContainerStarted","Data":"26875a1d1654c173114044a568f26e47db79d76a443d37dc2bb3b8f2d6dee495"} Oct 30 18:30:11.184668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:11.153896 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg" event={"ID":"80914a10-e845-4294-9423-7d356e84d57b","Type":"ContainerStarted","Data":"6f6c751f67b1d7a755991b237be66ddc74e26090a0660d8bc5ebd524524525fa"} Oct 30 18:30:11.184668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:11.153935 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22"] Oct 30 18:30:11.311275 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:11.283419 4603 scope.go:122] "RemoveContainer" containerID="6f6c751f67b1d7a755991b237be66ddc74e26090a0660d8bc5ebd524524525fa" Oct 30 18:30:11.311275 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:11.296133 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:30:11.353414 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:11.353344 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q"] Oct 30 18:30:11.677868 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:11.629129 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:11.677868 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:11.630074 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:11.677868 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:11.634637 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:11.677868 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:11.671668 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:11.677868 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:11.672036 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:30:12.109535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:12.101912 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:30:12.122720 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:12.122626 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:30:12.405970 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:12.396610 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:30:12.552532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:12.434715 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:30:12.688022 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:12.570916 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw"] Oct 30 18:30:12.688022 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:12.634743 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:12.756542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:12.711018 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:12.843491 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:12.843301 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:12.852538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:12.844966 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:30:12.852538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:12.846169 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:30:12.867551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:12.864389 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:30:12.905814 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:12.905770 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj"] Oct 30 18:30:12.942859 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:12.942770 4603 generic.go:356] "Generic (PLEG): container finished" podID="cf18bbde-1895-48b6-9ad1-7f6560660f8b" containerID="3eaf191ab371afbb4e52edafc000411222d0d023e0017dde3d9f7c6eb4ff0952" exitCode=60 Oct 30 18:30:12.943213 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:12.943193 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22" event={"ID":"cf18bbde-1895-48b6-9ad1-7f6560660f8b","Type":"ContainerDied","Data":"3eaf191ab371afbb4e52edafc000411222d0d023e0017dde3d9f7c6eb4ff0952"} Oct 30 18:30:12.943841 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:12.943822 4603 scope.go:122] "RemoveContainer" containerID="3eaf191ab371afbb4e52edafc000411222d0d023e0017dde3d9f7c6eb4ff0952" Oct 30 18:30:13.107078 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:13.103626 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:13.540548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:13.494202 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:13.581569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:13.574445 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:13.624565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:13.585225 4603 generic.go:356] "Generic (PLEG): container finished" podID="80914a10-e845-4294-9423-7d356e84d57b" containerID="6f6c751f67b1d7a755991b237be66ddc74e26090a0660d8bc5ebd524524525fa" exitCode=60 Oct 30 18:30:13.624565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:13.585366 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg" event={"ID":"80914a10-e845-4294-9423-7d356e84d57b","Type":"ContainerDied","Data":"6f6c751f67b1d7a755991b237be66ddc74e26090a0660d8bc5ebd524524525fa"} Oct 30 18:30:13.846628 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:13.846527 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:13.856901 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:13.856858 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:13.887722 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:13.864615 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:30:13.887722 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:13.876579 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:13.962820 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:13.956027 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:13.963527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:13.963474 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:14.029457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:13.982071 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22" podStartSLOduration=13.982038979 podStartE2EDuration="13.982038979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:12.134225271 +0000 UTC m=+2272.849795145" watchObservedRunningTime="2026-10-30 18:30:13.982038979 +0000 UTC m=+2274.697608854" Oct 30 18:30:14.110583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:14.087929 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:14.173699 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:14.173634 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:30:14.424591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:14.409331 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:14.565162 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:14.565104 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:14.657629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:14.657551 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:14.686926 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:14.678312 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:14.807959 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:14.807869 4603 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80914a10_e845_4294_9423_7d356e84d57b.slice/crio-6f6c751f67b1d7a755991b237be66ddc74e26090a0660d8bc5ebd524524525fa.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf18bbde_1895_48b6_9ad1_7f6560660f8b.slice/crio-conmon-3eaf191ab371afbb4e52edafc000411222d0d023e0017dde3d9f7c6eb4ff0952.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf18bbde_1895_48b6_9ad1_7f6560660f8b.slice/crio-3eaf191ab371afbb4e52edafc000411222d0d023e0017dde3d9f7c6eb4ff0952.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80914a10_e845_4294_9423_7d356e84d57b.slice/crio-conmon-6f6c751f67b1d7a755991b237be66ddc74e26090a0660d8bc5ebd524524525fa.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:30:14.868291 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:14.868248 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:14.882531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:14.878759 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:14.953016 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:14.952923 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:15.135409 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:14.960045 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:15.135938 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:15.026597 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:15.167692 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:15.059014 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:15.183549 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:15.061139 4603 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.424s" Oct 30 18:30:15.183549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:15.174722 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj" event={"ID":"9efc340c-4b4f-4a7f-ace1-0800c843361a","Type":"ContainerStarted","Data":"d639e574e18536566c80719e78bd1eb3849c8357b8b3348faf1ebeb155b12b22"} Oct 30 18:30:15.183549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:15.174891 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:30:15.183549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:15.174934 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:30:15.183549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:15.174982 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:30:15.183549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:15.175007 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc"] Oct 30 18:30:15.520581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:15.519642 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:30:15.570212 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:15.570145 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:15.588240 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:15.588200 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Oct 30 18:30:15.924960 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:15.905047 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:16.360525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:16.355211 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q" podStartSLOduration=16.355184357 podStartE2EDuration="16.355184357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:13.12834481 +0000 UTC m=+2273.843914684" watchObservedRunningTime="2026-10-30 18:30:16.355184357 +0000 UTC m=+2277.070754254" Oct 30 18:30:16.486841 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:16.486726 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:16.600604 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:16.571942 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:16.601415 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:16.601391 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:30:16.601693 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:16.601679 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:30:16.741696 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:16.741608 4603 generic.go:356] "Generic (PLEG): container finished" podID="be072889-51a1-4341-8559-6f445961c745" containerID="e489fcc57055b0e778f10931ae29fb51d899cf61fbbd113c85041fab8fd842af" exitCode=60 Oct 30 18:30:16.833524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:16.757653 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw" event={"ID":"be072889-51a1-4341-8559-6f445961c745","Type":"ContainerDied","Data":"e489fcc57055b0e778f10931ae29fb51d899cf61fbbd113c85041fab8fd842af"} Oct 30 18:30:16.833524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:16.758420 4603 scope.go:122] "RemoveContainer" containerID="e489fcc57055b0e778f10931ae29fb51d899cf61fbbd113c85041fab8fd842af" Oct 30 18:30:16.883978 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:16.883932 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf"] Oct 30 18:30:16.895542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:16.885963 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:16.955532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:16.941773 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:30:17.056609 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:17.056563 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:30:17.271442 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:17.271403 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:30:17.271781 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:17.271762 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:30:17.336111 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:17.334718 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:17.336111 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:17.335691 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:30:17.382138 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:17.382114 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:30:17.481449 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:17.481411 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:17.786088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:17.786022 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:30:17.886608 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:17.886462 4603 generic.go:356] "Generic (PLEG): container finished" podID="c5184124-242d-445e-9341-59d13045fda9" containerID="deb35bc736a856a28a5f6941e9db2f952dff0c2e1813cd06669d20b2362203d1" exitCode=60 Oct 30 18:30:17.887048 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:17.887018 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q" event={"ID":"c5184124-242d-445e-9341-59d13045fda9","Type":"ContainerDied","Data":"deb35bc736a856a28a5f6941e9db2f952dff0c2e1813cd06669d20b2362203d1"} Oct 30 18:30:18.159267 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:18.159181 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:18.160355 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:18.160325 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc" event={"ID":"103f4661-1a62-4f4d-8089-3ceabd3271ad","Type":"ContainerStarted","Data":"fd8c23b3de991f13d97b41d43707c0a82f1bbf09b8c9fae70b469fa833b5ec53"} Oct 30 18:30:18.160854 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:18.160834 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc" Oct 30 18:30:18.184532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:18.178471 4603 scope.go:122] "RemoveContainer" containerID="deb35bc736a856a28a5f6941e9db2f952dff0c2e1813cd06669d20b2362203d1" Oct 30 18:30:18.306604 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:18.290012 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg" event={"ID":"80914a10-e845-4294-9423-7d356e84d57b","Type":"ContainerStarted","Data":"03bbb14d6e24b32d120ae9d7e7728451421e1a704a8cb17440b4617cec4afb86"} Oct 30 18:30:18.312540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:18.308196 4603 scope.go:122] "RemoveContainer" containerID="03bbb14d6e24b32d120ae9d7e7728451421e1a704a8cb17440b4617cec4afb86" Oct 30 18:30:18.312540 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:18.308672 4603 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg_odh-ai-gateway-infra(80914a10-e845-4294-9423-7d356e84d57b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg" podUID="80914a10-e845-4294-9423-7d356e84d57b" Oct 30 18:30:18.379607 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:18.379574 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:18.387237 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:18.387207 4603 generic.go:356] "Generic (PLEG): container finished" podID="9efc340c-4b4f-4a7f-ace1-0800c843361a" containerID="22e70cc12948512513994651207ea1bbc58101d042aa9f92a3fbff9da5fc1117" exitCode=60 Oct 30 18:30:18.388085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:18.388001 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj" event={"ID":"9efc340c-4b4f-4a7f-ace1-0800c843361a","Type":"ContainerDied","Data":"22e70cc12948512513994651207ea1bbc58101d042aa9f92a3fbff9da5fc1117"} Oct 30 18:30:18.533996 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:18.505760 4603 scope.go:122] "RemoveContainer" containerID="22e70cc12948512513994651207ea1bbc58101d042aa9f92a3fbff9da5fc1117" Oct 30 18:30:18.843037 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:18.842999 4603 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9b49ca_d057_4d90_bab8_1519401aa457.slice/crio-conmon-45bda4e54b224495b141147eac4e025ed12cee8d9261269659d42c2ab644975f.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:30:18.897613 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:18.877828 4603 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9b49ca_d057_4d90_bab8_1519401aa457.slice/crio-conmon-45bda4e54b224495b141147eac4e025ed12cee8d9261269659d42c2ab644975f.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:30:19.003528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:18.978661 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:19.178994 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:19.178902 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc" podStartSLOduration=16.178882825 podStartE2EDuration="16.178882825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:18.648479379 +0000 UTC m=+2279.364049292" watchObservedRunningTime="2026-10-30 18:30:19.178882825 +0000 UTC m=+2279.894452704" Oct 30 18:30:19.459725 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:19.459672 4603 generic.go:356] "Generic (PLEG): container finished" podID="80914a10-e845-4294-9423-7d356e84d57b" containerID="03bbb14d6e24b32d120ae9d7e7728451421e1a704a8cb17440b4617cec4afb86" exitCode=60 Oct 30 18:30:19.460212 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:19.460177 4603 scope.go:122] "RemoveContainer" containerID="03bbb14d6e24b32d120ae9d7e7728451421e1a704a8cb17440b4617cec4afb86" Oct 30 18:30:19.460536 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:19.460462 4603 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg_odh-ai-gateway-infra(80914a10-e845-4294-9423-7d356e84d57b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg" podUID="80914a10-e845-4294-9423-7d356e84d57b" Oct 30 18:30:19.462870 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:19.462831 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg" event={"ID":"80914a10-e845-4294-9423-7d356e84d57b","Type":"ContainerDied","Data":"03bbb14d6e24b32d120ae9d7e7728451421e1a704a8cb17440b4617cec4afb86"} Oct 30 18:30:19.462952 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:19.462885 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf" event={"ID":"cb9b49ca-d057-4d90-bab8-1519401aa457","Type":"ContainerStarted","Data":"45bda4e54b224495b141147eac4e025ed12cee8d9261269659d42c2ab644975f"} Oct 30 18:30:19.462952 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:19.462908 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf" Oct 30 18:30:19.462952 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:19.462921 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf" event={"ID":"cb9b49ca-d057-4d90-bab8-1519401aa457","Type":"ContainerStarted","Data":"0f1f64c78cdc74497d966ac2d2c5c531c0fb43b8903976d55b19b4adc2a1946d"} Oct 30 18:30:19.462952 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:19.462947 4603 scope.go:122] "RemoveContainer" containerID="6f6c751f67b1d7a755991b237be66ddc74e26090a0660d8bc5ebd524524525fa" Oct 30 18:30:19.468135 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:19.468109 4603 generic.go:356] "Generic (PLEG): container finished" podID="be072889-51a1-4341-8559-6f445961c745" containerID="61f91e1ba75c1b5bd1bc362ea69c1a580a4a5aecb0dc64e86e91f0b98fd7090a" exitCode=60 Oct 30 18:30:19.468321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:19.468254 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw" event={"ID":"be072889-51a1-4341-8559-6f445961c745","Type":"ContainerDied","Data":"61f91e1ba75c1b5bd1bc362ea69c1a580a4a5aecb0dc64e86e91f0b98fd7090a"} Oct 30 18:30:19.468829 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:19.468679 4603 scope.go:122] "RemoveContainer" containerID="61f91e1ba75c1b5bd1bc362ea69c1a580a4a5aecb0dc64e86e91f0b98fd7090a" Oct 30 18:30:19.469191 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:19.468991 4603 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw_odh-ai-gateway-infra(be072889-51a1-4341-8559-6f445961c745)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw" podUID="be072889-51a1-4341-8559-6f445961c745" Oct 30 18:30:19.477409 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:19.477385 4603 generic.go:356] "Generic (PLEG): container finished" podID="cf18bbde-1895-48b6-9ad1-7f6560660f8b" containerID="8bc08a12821fe106c71eaf0f50030c8a421757b5689e7807152e70d089b90848" exitCode=60 Oct 30 18:30:19.477550 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:19.477536 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22" event={"ID":"cf18bbde-1895-48b6-9ad1-7f6560660f8b","Type":"ContainerDied","Data":"8bc08a12821fe106c71eaf0f50030c8a421757b5689e7807152e70d089b90848"} Oct 30 18:30:19.477850 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:19.477834 4603 scope.go:122] "RemoveContainer" containerID="8bc08a12821fe106c71eaf0f50030c8a421757b5689e7807152e70d089b90848" Oct 30 18:30:19.478084 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:19.478065 4603 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22_odh-ai-gateway-infra(cf18bbde-1895-48b6-9ad1-7f6560660f8b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22" podUID="cf18bbde-1895-48b6-9ad1-7f6560660f8b" Oct 30 18:30:19.481680 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:19.481614 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc" event={"ID":"103f4661-1a62-4f4d-8089-3ceabd3271ad","Type":"ContainerStarted","Data":"da804c96d6cd9673a8e34be3b42b5c9647cd5db7f594eabbee1ab4d353bcc31c"} Oct 30 18:30:19.483169 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:19.483149 4603 scope.go:122] "RemoveContainer" containerID="e489fcc57055b0e778f10931ae29fb51d899cf61fbbd113c85041fab8fd842af" Oct 30 18:30:19.536983 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:19.536965 4603 scope.go:122] "RemoveContainer" containerID="3eaf191ab371afbb4e52edafc000411222d0d023e0017dde3d9f7c6eb4ff0952" Oct 30 18:30:19.626516 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:19.626449 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf" podStartSLOduration=16.626429492 podStartE2EDuration="16.626429492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:19.562192064 +0000 UTC m=+2280.277761939" watchObservedRunningTime="2026-10-30 18:30:19.626429492 +0000 UTC m=+2280.341999405" Oct 30 18:30:19.719735 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:19.719695 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:20.522350 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:20.522320 4603 scope.go:122] "RemoveContainer" containerID="8bc08a12821fe106c71eaf0f50030c8a421757b5689e7807152e70d089b90848" Oct 30 18:30:20.522851 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:20.522748 4603 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22_odh-ai-gateway-infra(cf18bbde-1895-48b6-9ad1-7f6560660f8b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22" podUID="cf18bbde-1895-48b6-9ad1-7f6560660f8b" Oct 30 18:30:20.524441 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:20.524418 4603 generic.go:356] "Generic (PLEG): container finished" podID="c5184124-242d-445e-9341-59d13045fda9" containerID="fb9a3a574e6521b811caaee6655e56ba0c899b8fda33e54bd057a6619b375f2e" exitCode=60 Oct 30 18:30:20.524617 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:20.524493 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q" event={"ID":"c5184124-242d-445e-9341-59d13045fda9","Type":"ContainerDied","Data":"fb9a3a574e6521b811caaee6655e56ba0c899b8fda33e54bd057a6619b375f2e"} Oct 30 18:30:20.524617 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:20.524591 4603 scope.go:122] "RemoveContainer" containerID="deb35bc736a856a28a5f6941e9db2f952dff0c2e1813cd06669d20b2362203d1" Oct 30 18:30:20.525063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:20.525042 4603 scope.go:122] "RemoveContainer" containerID="fb9a3a574e6521b811caaee6655e56ba0c899b8fda33e54bd057a6619b375f2e" Oct 30 18:30:20.525461 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:20.525423 4603 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889750-v4r4q_odh-ai-gateway-infra(c5184124-242d-445e-9341-59d13045fda9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q" podUID="c5184124-242d-445e-9341-59d13045fda9" Oct 30 18:30:20.526593 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:20.526564 4603 scope.go:122] "RemoveContainer" containerID="03bbb14d6e24b32d120ae9d7e7728451421e1a704a8cb17440b4617cec4afb86" Oct 30 18:30:20.526921 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:20.526901 4603 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg_odh-ai-gateway-infra(80914a10-e845-4294-9423-7d356e84d57b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg" podUID="80914a10-e845-4294-9423-7d356e84d57b" Oct 30 18:30:20.528427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:20.528405 4603 generic.go:356] "Generic (PLEG): container finished" podID="9efc340c-4b4f-4a7f-ace1-0800c843361a" containerID="7ddaaf65838f60f47b2b1be7d7bd7127697c5de374256f9c3442653ffee30c5e" exitCode=60 Oct 30 18:30:20.528555 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:20.528492 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj" event={"ID":"9efc340c-4b4f-4a7f-ace1-0800c843361a","Type":"ContainerDied","Data":"7ddaaf65838f60f47b2b1be7d7bd7127697c5de374256f9c3442653ffee30c5e"} Oct 30 18:30:20.528964 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:20.528945 4603 scope.go:122] "RemoveContainer" containerID="7ddaaf65838f60f47b2b1be7d7bd7127697c5de374256f9c3442653ffee30c5e" Oct 30 18:30:20.529364 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:20.529324 4603 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj_odh-ai-gateway-infra(9efc340c-4b4f-4a7f-ace1-0800c843361a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj" podUID="9efc340c-4b4f-4a7f-ace1-0800c843361a" Oct 30 18:30:20.530787 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:20.530766 4603 scope.go:122] "RemoveContainer" containerID="61f91e1ba75c1b5bd1bc362ea69c1a580a4a5aecb0dc64e86e91f0b98fd7090a" Oct 30 18:30:20.531070 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:20.531050 4603 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw_odh-ai-gateway-infra(be072889-51a1-4341-8559-6f445961c745)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw" podUID="be072889-51a1-4341-8559-6f445961c745" Oct 30 18:30:20.538908 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:20.538891 4603 scope.go:122] "RemoveContainer" containerID="22e70cc12948512513994651207ea1bbc58101d042aa9f92a3fbff9da5fc1117" Oct 30 18:30:21.428539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:21.427621 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:21.571538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:21.571300 4603 scope.go:122] "RemoveContainer" containerID="7ddaaf65838f60f47b2b1be7d7bd7127697c5de374256f9c3442653ffee30c5e" Oct 30 18:30:21.572036 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:21.571778 4603 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj_odh-ai-gateway-infra(9efc340c-4b4f-4a7f-ace1-0800c843361a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj" podUID="9efc340c-4b4f-4a7f-ace1-0800c843361a" Oct 30 18:30:21.937323 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:21.937250 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:30:22.012718 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:22.012643 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:30:22.012880 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:22.012726 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" podUID="0baa3ccf-73e5-4555-9e51-5c627b74124d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:22.660553 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:22.660483 4603 scope.go:122] "RemoveContainer" containerID="927bde0819ecda487e1d05e1c7ec9ecb1f29a963e8991c98d10f681171bb61e1" Oct 30 18:30:23.617330 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:23.617237 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:23.774735 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:23.774690 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:30:23.775197 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:23.774766 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" podUID="a2d95ad6-c697-48a0-8592-0ff95ce3767b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:25.293445 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:25.293391 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:26.183634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:25.452435 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:26.183634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:25.651645 4603 generic.go:356] "Generic (PLEG): container finished" podID="0baa3ccf-73e5-4555-9e51-5c627b74124d" containerID="e627332c30048dc62c7abd519e19bafc165413f069b36427fb4456800e4971e9" exitCode=0 Oct 30 18:30:26.183634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:25.651681 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" event={"ID":"0baa3ccf-73e5-4555-9e51-5c627b74124d","Type":"ContainerDied","Data":"e627332c30048dc62c7abd519e19bafc165413f069b36427fb4456800e4971e9"} Oct 30 18:30:26.183634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:25.652093 4603 scope.go:122] "RemoveContainer" containerID="e627332c30048dc62c7abd519e19bafc165413f069b36427fb4456800e4971e9" Oct 30 18:30:26.183634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:25.653882 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:26.183634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:25.653955 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:26.183634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:25.653985 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:26.406529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:26.404850 4603 scope.go:122] "RemoveContainer" containerID="03bbb14d6e24b32d120ae9d7e7728451421e1a704a8cb17440b4617cec4afb86" Oct 30 18:30:26.660769 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:26.660738 4603 generic.go:356] "Generic (PLEG): container finished" podID="a2d95ad6-c697-48a0-8592-0ff95ce3767b" containerID="69c0db244afc76ed20a51a416413b30baf9462d999fb95e2b0b5f000b1f60a21" exitCode=0 Oct 30 18:30:26.660953 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:26.660818 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" event={"ID":"a2d95ad6-c697-48a0-8592-0ff95ce3767b","Type":"ContainerDied","Data":"69c0db244afc76ed20a51a416413b30baf9462d999fb95e2b0b5f000b1f60a21"} Oct 30 18:30:26.661312 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:26.661293 4603 scope.go:122] "RemoveContainer" containerID="69c0db244afc76ed20a51a416413b30baf9462d999fb95e2b0b5f000b1f60a21" Oct 30 18:30:26.662986 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:26.662963 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" event={"ID":"0baa3ccf-73e5-4555-9e51-5c627b74124d","Type":"ContainerStarted","Data":"8f34566063990fb14ef493b7fa4d7f960c042d04c8b9d2d43d0d2c316bcecfec"} Oct 30 18:30:26.663237 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:26.663195 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:26.663353 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:26.663286 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:26.663353 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:26.663331 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:26.664741 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:26.664722 4603 generic.go:356] "Generic (PLEG): container finished" podID="dba942a9-7d94-415a-8491-a17212be1601" containerID="38a01e2bc6fb4ba12131c55003733c74e3fff4f17fb808240979c542a9be32b4" exitCode=0 Oct 30 18:30:26.664821 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:26.664793 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" event={"ID":"dba942a9-7d94-415a-8491-a17212be1601","Type":"ContainerDied","Data":"38a01e2bc6fb4ba12131c55003733c74e3fff4f17fb808240979c542a9be32b4"} Oct 30 18:30:26.665240 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:26.665223 4603 scope.go:122] "RemoveContainer" containerID="38a01e2bc6fb4ba12131c55003733c74e3fff4f17fb808240979c542a9be32b4" Oct 30 18:30:26.667029 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:26.666943 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:26.667029 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:26.667005 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:26.667118 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:26.667040 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:27.426521 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:27.424619 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 18:30:27.426521 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:27.424681 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" podUID="0baa3ccf-73e5-4555-9e51-5c627b74124d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 18:30:27.426521 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:27.426269 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:27.552528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:27.546006 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:30:27.552528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:27.546721 4603 scope.go:122] "RemoveContainer" containerID="8bc08a12821fe106c71eaf0f50030c8a421757b5689e7807152e70d089b90848" Oct 30 18:30:27.743533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:27.743006 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:27.763738 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:27.754821 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" event={"ID":"a2d95ad6-c697-48a0-8592-0ff95ce3767b","Type":"ContainerStarted","Data":"24d8106b8305624ad78d84f82882a972234217dd577d3d34c1d6aa6f65ac9af1"} Oct 30 18:30:27.763738 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:27.759777 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" event={"ID":"dba942a9-7d94-415a-8491-a17212be1601","Type":"ContainerStarted","Data":"81f1036c94bcd2e63a7c2329b4824e4d4f21a71c9904da97d891d82f12060fda"} Oct 30 18:30:27.768837 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:27.768371 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg" event={"ID":"80914a10-e845-4294-9423-7d356e84d57b","Type":"ContainerStarted","Data":"31420881064cdd683b99aa29fb9cc8dc25c2defc1a1976fb1a6af39e1a4ac905"} Oct 30 18:30:27.769026 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:27.768999 4603 scope.go:122] "RemoveContainer" containerID="31420881064cdd683b99aa29fb9cc8dc25c2defc1a1976fb1a6af39e1a4ac905" Oct 30 18:30:27.769333 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:27.769310 4603 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg_odh-ai-gateway-infra(80914a10-e845-4294-9423-7d356e84d57b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg" podUID="80914a10-e845-4294-9423-7d356e84d57b" Oct 30 18:30:27.988197 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:27.988109 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:28.040649 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:28.040612 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:30:28.041428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:28.040743 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 18:30:28.041428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:28.040809 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" podUID="dba942a9-7d94-415a-8491-a17212be1601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 18:30:28.238621 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:28.217964 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:28.397530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:28.396880 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 18:30:28.397530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:28.396939 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" podUID="0baa3ccf-73e5-4555-9e51-5c627b74124d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 18:30:28.410528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:28.409274 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:28.699332 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:28.698908 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:30:28.699332 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:28.699029 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 18:30:28.699332 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:28.699071 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" podUID="a2d95ad6-c697-48a0-8592-0ff95ce3767b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 18:30:28.757357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:28.757153 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:28.786314 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:28.786278 4603 generic.go:356] "Generic (PLEG): container finished" podID="80914a10-e845-4294-9423-7d356e84d57b" containerID="31420881064cdd683b99aa29fb9cc8dc25c2defc1a1976fb1a6af39e1a4ac905" exitCode=60 Oct 30 18:30:28.786559 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:28.786340 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg" event={"ID":"80914a10-e845-4294-9423-7d356e84d57b","Type":"ContainerDied","Data":"31420881064cdd683b99aa29fb9cc8dc25c2defc1a1976fb1a6af39e1a4ac905"} Oct 30 18:30:28.786559 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:28.786401 4603 scope.go:122] "RemoveContainer" containerID="03bbb14d6e24b32d120ae9d7e7728451421e1a704a8cb17440b4617cec4afb86" Oct 30 18:30:28.800533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:28.795100 4603 scope.go:122] "RemoveContainer" containerID="31420881064cdd683b99aa29fb9cc8dc25c2defc1a1976fb1a6af39e1a4ac905" Oct 30 18:30:28.800533 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:28.795554 4603 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg_odh-ai-gateway-infra(80914a10-e845-4294-9423-7d356e84d57b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg" podUID="80914a10-e845-4294-9423-7d356e84d57b" Oct 30 18:30:28.827530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:28.825684 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22" event={"ID":"cf18bbde-1895-48b6-9ad1-7f6560660f8b","Type":"ContainerStarted","Data":"018e4eda0f35cf24ca633dd62a93a1d64e104b93ba6b67d6e971a4c9c3cdf489"} Oct 30 18:30:29.024528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:29.024116 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 18:30:29.024528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:29.024180 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" podUID="dba942a9-7d94-415a-8491-a17212be1601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 18:30:29.072543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:29.063659 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg"] Oct 30 18:30:29.444203 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:29.420131 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 18:30:29.444203 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:29.420189 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" podUID="0baa3ccf-73e5-4555-9e51-5c627b74124d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 18:30:29.444203 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:29.439022 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:29.803530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:29.798115 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8"] Oct 30 18:30:29.869082 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:29.869044 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 18:30:29.869383 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:29.869353 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" podUID="a2d95ad6-c697-48a0-8592-0ff95ce3767b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 18:30:29.954790 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:29.880590 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:29.960337 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:29.960132 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:30.146583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.139700 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 18:30:30.146583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.139772 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" podUID="dba942a9-7d94-415a-8491-a17212be1601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 18:30:30.265694 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.264998 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:30.382008 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.322489 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:30.461435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.367870 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:30:30.461435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.398705 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8"] Oct 30 18:30:30.461435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.431048 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22"] Oct 30 18:30:30.461435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.398800 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.461435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.414992 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 18:30:30.461435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.432326 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" podUID="0baa3ccf-73e5-4555-9e51-5c627b74124d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 18:30:30.461435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.456160 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-3b8bf3-openshift-default-dockercfg-256pt\"" Oct 30 18:30:30.492900 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.479335 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwpbr\" (UniqueName: \"kubernetes.io/projected/73366d41-4910-43f0-aad8-2542401ec0f6-kube-api-access-dwpbr\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.493697 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.493289 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:30.579538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.557590 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73366d41-4910-43f0-aad8-2542401ec0f6-istio-podinfo\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.579538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.557770 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-workload-socket\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.579538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.557831 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73366d41-4910-43f0-aad8-2542401ec0f6-istio-token\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.579538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.557885 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-istio-envoy\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.579538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.557956 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-credential-socket\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.579538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.558007 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-workload-certs\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.579538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.558038 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-istio-data\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.579538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.558120 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73366d41-4910-43f0-aad8-2542401ec0f6-istiod-ca-cert\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.697639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.666787 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc" Oct 30 18:30:30.697639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.668933 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73366d41-4910-43f0-aad8-2542401ec0f6-istiod-ca-cert\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.697639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.669074 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwpbr\" (UniqueName: \"kubernetes.io/projected/73366d41-4910-43f0-aad8-2542401ec0f6-kube-api-access-dwpbr\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.697639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.669242 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73366d41-4910-43f0-aad8-2542401ec0f6-istio-podinfo\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.697639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.669467 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-workload-socket\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.697639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.669578 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73366d41-4910-43f0-aad8-2542401ec0f6-istio-token\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.697639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.669639 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-istio-envoy\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.697639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.669718 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-credential-socket\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.697639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.669767 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-workload-certs\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.697639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.669790 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-istio-data\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.697639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.679062 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-workload-socket\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.697639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.679821 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73366d41-4910-43f0-aad8-2542401ec0f6-istiod-ca-cert\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.778528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.751841 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf" Oct 30 18:30:30.808464 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.808427 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-credential-socket\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.864706 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.864654 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 18:30:30.865102 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.865054 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" podUID="a2d95ad6-c697-48a0-8592-0ff95ce3767b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 18:30:30.892572 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.885766 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-workload-certs\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.893355 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.893323 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:30.927551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.927487 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-istio-data\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:30.984579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.963705 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:30.984579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:30.975290 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp"] Oct 30 18:30:31.103771 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.091154 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73366d41-4910-43f0-aad8-2542401ec0f6-istio-podinfo\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:31.134145 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.134111 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:31.135474 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.135441 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwpbr\" (UniqueName: \"kubernetes.io/projected/73366d41-4910-43f0-aad8-2542401ec0f6-kube-api-access-dwpbr\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:31.183122 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.183077 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-istio-envoy\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:31.231570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.184805 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73366d41-4910-43f0-aad8-2542401ec0f6-istio-token\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:31.231570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.206555 4603 generic.go:356] "Generic (PLEG): container finished" podID="cf18bbde-1895-48b6-9ad1-7f6560660f8b" containerID="018e4eda0f35cf24ca633dd62a93a1d64e104b93ba6b67d6e971a4c9c3cdf489" exitCode=60 Oct 30 18:30:31.247575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.234006 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 18:30:31.247856 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.247811 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" podUID="dba942a9-7d94-415a-8491-a17212be1601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 18:30:31.284961 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.284931 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:31.288061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.288042 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:30:31.288324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.288237 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp" Oct 30 18:30:31.297428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.288290 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22" event={"ID":"cf18bbde-1895-48b6-9ad1-7f6560660f8b","Type":"ContainerDied","Data":"018e4eda0f35cf24ca633dd62a93a1d64e104b93ba6b67d6e971a4c9c3cdf489"} Oct 30 18:30:31.372611 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.372568 4603 scope.go:122] "RemoveContainer" containerID="8bc08a12821fe106c71eaf0f50030c8a421757b5689e7807152e70d089b90848" Oct 30 18:30:31.373747 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.373719 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp"] Oct 30 18:30:31.380080 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.380049 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc"] Oct 30 18:30:31.430536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.421881 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d99bf6c9-a08f-4f49-b845-0f411f1f1720-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp\" (UID: \"d99bf6c9-a08f-4f49-b845-0f411f1f1720\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp" Oct 30 18:30:31.492724 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.492686 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 18:30:31.493032 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.492995 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" podUID="0baa3ccf-73e5-4555-9e51-5c627b74124d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 18:30:31.598557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.595836 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfds6\" (UniqueName: \"kubernetes.io/projected/d99bf6c9-a08f-4f49-b845-0f411f1f1720-kube-api-access-tfds6\") pod \"payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp\" (UID: \"d99bf6c9-a08f-4f49-b845-0f411f1f1720\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp" Oct 30 18:30:31.674817 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.674787 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc" Oct 30 18:30:31.691752 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.691137 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 18:30:31.695530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.691998 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" podUID="a2d95ad6-c697-48a0-8592-0ff95ce3767b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 18:30:31.767938 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.722012 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfds6\" (UniqueName: \"kubernetes.io/projected/d99bf6c9-a08f-4f49-b845-0f411f1f1720-kube-api-access-tfds6\") pod \"payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp\" (UID: \"d99bf6c9-a08f-4f49-b845-0f411f1f1720\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp" Oct 30 18:30:31.767938 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.722347 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d99bf6c9-a08f-4f49-b845-0f411f1f1720-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp\" (UID: \"d99bf6c9-a08f-4f49-b845-0f411f1f1720\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp" Oct 30 18:30:31.767938 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.735711 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-c8eef3-dockercfg-9r6fs\"" Oct 30 18:30:31.767938 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.737113 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-c8eef3\"" Oct 30 18:30:31.767938 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.737223 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:31.808538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.795031 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d99bf6c9-a08f-4f49-b845-0f411f1f1720-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp\" (UID: \"d99bf6c9-a08f-4f49-b845-0f411f1f1720\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp" Oct 30 18:30:31.875832 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.871728 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjxn6\" (UniqueName: \"kubernetes.io/projected/c41e57c9-3c80-40fc-8ab4-8b3df57b0136-kube-api-access-tjxn6\") pod \"payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc\" (UID: \"c41e57c9-3c80-40fc-8ab4-8b3df57b0136\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc" Oct 30 18:30:31.875832 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.871800 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c41e57c9-3c80-40fc-8ab4-8b3df57b0136-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc\" (UID: \"c41e57c9-3c80-40fc-8ab4-8b3df57b0136\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc" Oct 30 18:30:31.904730 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.903262 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfds6\" (UniqueName: \"kubernetes.io/projected/d99bf6c9-a08f-4f49-b845-0f411f1f1720-kube-api-access-tfds6\") pod \"payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp\" (UID: \"d99bf6c9-a08f-4f49-b845-0f411f1f1720\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp" Oct 30 18:30:31.934597 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.934566 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:30:31.934856 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.934839 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc"] Oct 30 18:30:31.968593 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.968551 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:31.993038 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.993003 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp" Oct 30 18:30:31.997353 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.997312 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjxn6\" (UniqueName: \"kubernetes.io/projected/c41e57c9-3c80-40fc-8ab4-8b3df57b0136-kube-api-access-tjxn6\") pod \"payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc\" (UID: \"c41e57c9-3c80-40fc-8ab4-8b3df57b0136\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc" Oct 30 18:30:32.020582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:31.998686 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c41e57c9-3c80-40fc-8ab4-8b3df57b0136-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc\" (UID: \"c41e57c9-3c80-40fc-8ab4-8b3df57b0136\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc" Oct 30 18:30:32.020879 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:32.012571 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c41e57c9-3c80-40fc-8ab4-8b3df57b0136-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc\" (UID: \"c41e57c9-3c80-40fc-8ab4-8b3df57b0136\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc" Oct 30 18:30:32.020997 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:32.019563 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 18:30:32.021145 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:32.021120 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" podUID="dba942a9-7d94-415a-8491-a17212be1601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 18:30:32.022058 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:32.022012 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjxn6\" (UniqueName: \"kubernetes.io/projected/c41e57c9-3c80-40fc-8ab4-8b3df57b0136-kube-api-access-tjxn6\") pod \"payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc\" (UID: \"c41e57c9-3c80-40fc-8ab4-8b3df57b0136\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc" Oct 30 18:30:32.246588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:32.244187 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc" Oct 30 18:30:32.479756 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:32.427272 4603 scope.go:122] "RemoveContainer" containerID="fb9a3a574e6521b811caaee6655e56ba0c899b8fda33e54bd057a6619b375f2e" Oct 30 18:30:32.479756 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:32.438760 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:30:32.479756 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:32.439119 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:30:32.479756 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:32.441041 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 18:30:32.479756 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:32.441079 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" podUID="0baa3ccf-73e5-4555-9e51-5c627b74124d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 18:30:32.479756 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:32.441899 4603 scope.go:122] "RemoveContainer" containerID="61f91e1ba75c1b5bd1bc362ea69c1a580a4a5aecb0dc64e86e91f0b98fd7090a" Oct 30 18:30:32.479756 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:32.442779 4603 scope.go:122] "RemoveContainer" containerID="7ddaaf65838f60f47b2b1be7d7bd7127697c5de374256f9c3442653ffee30c5e" Oct 30 18:30:32.616670 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:32.616473 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:32.774713 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:32.747630 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 18:30:32.774713 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:32.747686 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" podUID="a2d95ad6-c697-48a0-8592-0ff95ce3767b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 18:30:32.800558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:32.775661 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:32.802352 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:32.801855 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:32.982634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:32.982579 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:33.061163 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:33.025603 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 18:30:33.061163 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:33.025657 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" podUID="dba942a9-7d94-415a-8491-a17212be1601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 18:30:33.367542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:33.367494 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:30:33.499805 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:33.499770 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 18:30:33.500616 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:33.500582 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" podUID="0baa3ccf-73e5-4555-9e51-5c627b74124d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 18:30:33.678571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:33.651330 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:30:33.746593 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:33.737964 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 18:30:33.746593 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:33.738027 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" podUID="a2d95ad6-c697-48a0-8592-0ff95ce3767b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 18:30:34.044903 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.044871 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 18:30:34.045889 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.045848 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" podUID="dba942a9-7d94-415a-8491-a17212be1601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 18:30:34.078726 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.074942 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22" Oct 30 18:30:34.081050 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.081030 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg" Oct 30 18:30:34.143725 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.143691 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf18bbde-1895-48b6-9ad1-7f6560660f8b-kube-api-access-dff87\" (UniqueName: \"kubernetes.io/projected/cf18bbde-1895-48b6-9ad1-7f6560660f8b-kube-api-access-dff87\") pod \"cf18bbde-1895-48b6-9ad1-7f6560660f8b\" (UID: \"cf18bbde-1895-48b6-9ad1-7f6560660f8b\") " Oct 30 18:30:34.164197 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.164164 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf18bbde-1895-48b6-9ad1-7f6560660f8b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf18bbde-1895-48b6-9ad1-7f6560660f8b-maas-api-service-ca\") pod \"cf18bbde-1895-48b6-9ad1-7f6560660f8b\" (UID: \"cf18bbde-1895-48b6-9ad1-7f6560660f8b\") " Oct 30 18:30:34.164483 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.164467 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80914a10-e845-4294-9423-7d356e84d57b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80914a10-e845-4294-9423-7d356e84d57b-maas-api-service-ca\") pod \"80914a10-e845-4294-9423-7d356e84d57b\" (UID: \"80914a10-e845-4294-9423-7d356e84d57b\") " Oct 30 18:30:34.164660 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.164645 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80914a10-e845-4294-9423-7d356e84d57b-kube-api-access-r6lz8\" (UniqueName: \"kubernetes.io/projected/80914a10-e845-4294-9423-7d356e84d57b-kube-api-access-r6lz8\") pod \"80914a10-e845-4294-9423-7d356e84d57b\" (UID: \"80914a10-e845-4294-9423-7d356e84d57b\") " Oct 30 18:30:34.172896 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.166717 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf18bbde-1895-48b6-9ad1-7f6560660f8b-maas-api-service-ca" pod "cf18bbde-1895-48b6-9ad1-7f6560660f8b" (UID: "cf18bbde-1895-48b6-9ad1-7f6560660f8b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:34.172896 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.167061 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80914a10-e845-4294-9423-7d356e84d57b-maas-api-service-ca" pod "80914a10-e845-4294-9423-7d356e84d57b" (UID: "80914a10-e845-4294-9423-7d356e84d57b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:34.172896 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.170619 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22" event={"ID":"cf18bbde-1895-48b6-9ad1-7f6560660f8b","Type":"ContainerDied","Data":"4206f866113caa22bf41ea465179a81ba349e15ea906c129ee2be1529aed56d3"} Oct 30 18:30:34.172896 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.170726 4603 scope.go:122] "RemoveContainer" containerID="018e4eda0f35cf24ca633dd62a93a1d64e104b93ba6b67d6e971a4c9c3cdf489" Oct 30 18:30:34.172896 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.170912 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22" Oct 30 18:30:34.173778 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.173759 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg" Oct 30 18:30:34.174215 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.174194 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg" event={"ID":"80914a10-e845-4294-9423-7d356e84d57b","Type":"ContainerDied","Data":"48960a248f2439b5b385f302de2050fe7e3fdcf862c2e3fe27be829b3a794962"} Oct 30 18:30:34.178796 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.178740 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf18bbde-1895-48b6-9ad1-7f6560660f8b-kube-api-access-dff87" pod "cf18bbde-1895-48b6-9ad1-7f6560660f8b" (UID: "cf18bbde-1895-48b6-9ad1-7f6560660f8b"). InnerVolumeSpecName "kube-api-access-dff87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:34.181642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.181612 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80914a10-e845-4294-9423-7d356e84d57b-kube-api-access-r6lz8" pod "80914a10-e845-4294-9423-7d356e84d57b" (UID: "80914a10-e845-4294-9423-7d356e84d57b"). InnerVolumeSpecName "kube-api-access-r6lz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:34.270226 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.270105 4603 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80914a10-e845-4294-9423-7d356e84d57b-maas-api-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:30:34.270226 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.270144 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6lz8\" (UniqueName: \"kubernetes.io/projected/80914a10-e845-4294-9423-7d356e84d57b-kube-api-access-r6lz8\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:30:34.270226 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.270162 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dff87\" (UniqueName: \"kubernetes.io/projected/cf18bbde-1895-48b6-9ad1-7f6560660f8b-kube-api-access-dff87\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:30:34.270226 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.270178 4603 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf18bbde-1895-48b6-9ad1-7f6560660f8b-maas-api-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:30:34.504784 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.483033 4603 scope.go:122] "RemoveContainer" containerID="31420881064cdd683b99aa29fb9cc8dc25c2defc1a1976fb1a6af39e1a4ac905" Oct 30 18:30:34.626147 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.511943 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:34.682728 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.682707 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp"] Oct 30 18:30:34.717131 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.717100 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 18:30:34.752486 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.752455 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" podUID="a2d95ad6-c697-48a0-8592-0ff95ce3767b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 18:30:34.753927 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.753904 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8"] Oct 30 18:30:34.761902 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.761876 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:30:34.764099 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.764080 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:30:34.765713 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.765474 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:34.765713 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.765599 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:34.765713 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.765643 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:34.771402 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.770312 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:30:34.883769 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.882561 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22"] Oct 30 18:30:34.924527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.887471 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e77b1d-29889750-vhs22"] Oct 30 18:30:34.987689 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:34.987040 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg"] Oct 30 18:30:35.026528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.024360 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 18:30:35.026528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.024538 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" podUID="dba942a9-7d94-415a-8491-a17212be1601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 18:30:35.071091 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.070987 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889750-jcqbg"] Oct 30 18:30:35.076567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.076488 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc"] Oct 30 18:30:35.221357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.221324 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp" event={"ID":"d99bf6c9-a08f-4f49-b845-0f411f1f1720","Type":"ContainerStarted","Data":"34b3f2d7c8d7ee0152a6cb46bf226ddef3f0388233e6f727e8a9a694816c812d"} Oct 30 18:30:35.221633 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.221608 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp" event={"ID":"d99bf6c9-a08f-4f49-b845-0f411f1f1720","Type":"ContainerStarted","Data":"e446a93412ab31ec14a8850fb008a97135b956ba0e85986e6ce8d714866b44db"} Oct 30 18:30:35.222994 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.222972 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp" Oct 30 18:30:35.268783 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.267802 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q" event={"ID":"c5184124-242d-445e-9341-59d13045fda9","Type":"ContainerStarted","Data":"4904b7e53a1ae60bab32e566588a4d8dd5d1ab0b6913fc2f1fd0a4eca100f480"} Oct 30 18:30:35.270165 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.270135 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" event={"ID":"73366d41-4910-43f0-aad8-2542401ec0f6","Type":"ContainerStarted","Data":"d95c310ab51db3b9a868be677d78cc335b2b18e6ac980e855881ec4e704755d7"} Oct 30 18:30:35.270309 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.270188 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" event={"ID":"73366d41-4910-43f0-aad8-2542401ec0f6","Type":"ContainerStarted","Data":"19ddfd1635b62dd96160a04887e1a6b982cc9f8bd892dce48c91772adac8bd1f"} Oct 30 18:30:35.275726 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.275698 4603 generic.go:356] "Generic (PLEG): container finished" podID="9efc340c-4b4f-4a7f-ace1-0800c843361a" containerID="4b89546e3b3473862ba9909b695572a8b02687757a8562abe063141600e0b807" exitCode=60 Oct 30 18:30:35.275841 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.275791 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj" event={"ID":"9efc340c-4b4f-4a7f-ace1-0800c843361a","Type":"ContainerDied","Data":"4b89546e3b3473862ba9909b695572a8b02687757a8562abe063141600e0b807"} Oct 30 18:30:35.275841 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.275832 4603 scope.go:122] "RemoveContainer" containerID="7ddaaf65838f60f47b2b1be7d7bd7127697c5de374256f9c3442653ffee30c5e" Oct 30 18:30:35.276316 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.276282 4603 scope.go:122] "RemoveContainer" containerID="4b89546e3b3473862ba9909b695572a8b02687757a8562abe063141600e0b807" Oct 30 18:30:35.276656 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:35.276632 4603 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj_odh-ai-gateway-infra(9efc340c-4b4f-4a7f-ace1-0800c843361a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj" podUID="9efc340c-4b4f-4a7f-ace1-0800c843361a" Oct 30 18:30:35.278517 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.277776 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc" event={"ID":"c41e57c9-3c80-40fc-8ab4-8b3df57b0136","Type":"ContainerStarted","Data":"4b56ce2969b8fea8f285ece31afc96ee5a40dd190a3a25748e056ef2bd6fff71"} Oct 30 18:30:35.280863 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.280790 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw" event={"ID":"be072889-51a1-4341-8559-6f445961c745","Type":"ContainerStarted","Data":"eaa3c8cfc11b1ba9fa4658831962da277cbda19dd2274af5ea4fd0cf81f203a6"} Oct 30 18:30:35.285273 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.285253 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:35.312380 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.312183 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp" podStartSLOduration=5.312160362 podStartE2EDuration="5.312160362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:35.299770164 +0000 UTC m=+2296.015340068" watchObservedRunningTime="2026-10-30 18:30:35.312160362 +0000 UTC m=+2296.027730237" Oct 30 18:30:35.389468 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.389407 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw" podStartSLOduration=35.389387813 podStartE2EDuration="35.389387813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:35.386010022 +0000 UTC m=+2296.101579898" watchObservedRunningTime="2026-10-30 18:30:35.389387813 +0000 UTC m=+2296.104957690" Oct 30 18:30:35.518519 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.518384 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" podStartSLOduration=6.518363163 podStartE2EDuration="6.518363163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:35.501997061 +0000 UTC m=+2296.217566936" watchObservedRunningTime="2026-10-30 18:30:35.518363163 +0000 UTC m=+2296.233933038" Oct 30 18:30:35.555035 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.545153 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80914a10-e845-4294-9423-7d356e84d57b" path="/var/lib/kubelet/pods/80914a10-e845-4294-9423-7d356e84d57b/volumes" Oct 30 18:30:35.555035 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.545784 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf18bbde-1895-48b6-9ad1-7f6560660f8b" path="/var/lib/kubelet/pods/cf18bbde-1895-48b6-9ad1-7f6560660f8b/volumes" Oct 30 18:30:35.959722 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.959688 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:30:35.960334 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.960306 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:30:35.967596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.967571 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:30:35.974297 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:35.974276 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:30:36.027900 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.027873 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:30:36.290894 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.290855 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc" event={"ID":"c41e57c9-3c80-40fc-8ab4-8b3df57b0136","Type":"ContainerStarted","Data":"6fd54d6326384b91bfb34d85ecee4d31bef6d3b31056594675c175ef7512edf9"} Oct 30 18:30:36.291336 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.290928 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc" Oct 30 18:30:36.292421 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.292401 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:36.292937 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.292914 4603 generic.go:356] "Generic (PLEG): container finished" podID="be072889-51a1-4341-8559-6f445961c745" containerID="eaa3c8cfc11b1ba9fa4658831962da277cbda19dd2274af5ea4fd0cf81f203a6" exitCode=60 Oct 30 18:30:36.293044 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.292978 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw" event={"ID":"be072889-51a1-4341-8559-6f445961c745","Type":"ContainerDied","Data":"eaa3c8cfc11b1ba9fa4658831962da277cbda19dd2274af5ea4fd0cf81f203a6"} Oct 30 18:30:36.293044 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.293029 4603 scope.go:122] "RemoveContainer" containerID="61f91e1ba75c1b5bd1bc362ea69c1a580a4a5aecb0dc64e86e91f0b98fd7090a" Oct 30 18:30:36.293431 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.293413 4603 scope.go:122] "RemoveContainer" containerID="eaa3c8cfc11b1ba9fa4658831962da277cbda19dd2274af5ea4fd0cf81f203a6" Oct 30 18:30:36.293850 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:36.293824 4603 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw_odh-ai-gateway-infra(be072889-51a1-4341-8559-6f445961c745)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw" podUID="be072889-51a1-4341-8559-6f445961c745" Oct 30 18:30:36.295274 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.295254 4603 generic.go:356] "Generic (PLEG): container finished" podID="c5184124-242d-445e-9341-59d13045fda9" containerID="4904b7e53a1ae60bab32e566588a4d8dd5d1ab0b6913fc2f1fd0a4eca100f480" exitCode=60 Oct 30 18:30:36.295355 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.295323 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q" event={"ID":"c5184124-242d-445e-9341-59d13045fda9","Type":"ContainerDied","Data":"4904b7e53a1ae60bab32e566588a4d8dd5d1ab0b6913fc2f1fd0a4eca100f480"} Oct 30 18:30:36.295742 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.295727 4603 scope.go:122] "RemoveContainer" containerID="4904b7e53a1ae60bab32e566588a4d8dd5d1ab0b6913fc2f1fd0a4eca100f480" Oct 30 18:30:36.296047 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:36.296027 4603 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889750-v4r4q_odh-ai-gateway-infra(c5184124-242d-445e-9341-59d13045fda9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q" podUID="c5184124-242d-445e-9341-59d13045fda9" Oct 30 18:30:36.298206 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.298178 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:36.298302 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.298237 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:30:36.298983 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.298962 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:30:36.308099 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.308080 4603 scope.go:122] "RemoveContainer" containerID="fb9a3a574e6521b811caaee6655e56ba0c899b8fda33e54bd057a6619b375f2e" Oct 30 18:30:36.334948 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.334907 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc" podStartSLOduration=6.334896701 podStartE2EDuration="6.334896701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:36.332657936 +0000 UTC m=+2297.048227810" watchObservedRunningTime="2026-10-30 18:30:36.334896701 +0000 UTC m=+2297.050466604" Oct 30 18:30:36.343280 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.343259 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:36.354285 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.354258 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj"] Oct 30 18:30:36.403842 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.403752 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw"] Oct 30 18:30:36.456309 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.456279 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q"] Oct 30 18:30:36.662746 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.662466 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:36.685471 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.685441 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj" Oct 30 18:30:36.775562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.770344 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9efc340c-4b4f-4a7f-ace1-0800c843361a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9efc340c-4b4f-4a7f-ace1-0800c843361a-maas-api-service-ca\") pod \"9efc340c-4b4f-4a7f-ace1-0800c843361a\" (UID: \"9efc340c-4b4f-4a7f-ace1-0800c843361a\") " Oct 30 18:30:36.775562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.770412 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9efc340c-4b4f-4a7f-ace1-0800c843361a-kube-api-access-np4p5\" (UniqueName: \"kubernetes.io/projected/9efc340c-4b4f-4a7f-ace1-0800c843361a-kube-api-access-np4p5\") pod \"9efc340c-4b4f-4a7f-ace1-0800c843361a\" (UID: \"9efc340c-4b4f-4a7f-ace1-0800c843361a\") " Oct 30 18:30:36.777964 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.776051 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9efc340c-4b4f-4a7f-ace1-0800c843361a-maas-api-service-ca" pod "9efc340c-4b4f-4a7f-ace1-0800c843361a" (UID: "9efc340c-4b4f-4a7f-ace1-0800c843361a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:36.794525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.794077 4603 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9efc340c-4b4f-4a7f-ace1-0800c843361a-maas-api-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:30:36.867426 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.867333 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:36.870845 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.870099 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9efc340c-4b4f-4a7f-ace1-0800c843361a-kube-api-access-np4p5" pod "9efc340c-4b4f-4a7f-ace1-0800c843361a" (UID: "9efc340c-4b4f-4a7f-ace1-0800c843361a"). InnerVolumeSpecName "kube-api-access-np4p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:36.896004 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:36.895978 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np4p5\" (UniqueName: \"kubernetes.io/projected/9efc340c-4b4f-4a7f-ace1-0800c843361a-kube-api-access-np4p5\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:30:37.305995 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:37.305969 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj" Oct 30 18:30:37.306550 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:37.305998 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj" event={"ID":"9efc340c-4b4f-4a7f-ace1-0800c843361a","Type":"ContainerDied","Data":"d639e574e18536566c80719e78bd1eb3849c8357b8b3348faf1ebeb155b12b22"} Oct 30 18:30:37.306550 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:37.306072 4603 scope.go:122] "RemoveContainer" containerID="4b89546e3b3473862ba9909b695572a8b02687757a8562abe063141600e0b807" Oct 30 18:30:37.452836 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:37.452802 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj"] Oct 30 18:30:37.458747 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:37.458720 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-5cb3d5-29889750-jxfmj"] Oct 30 18:30:37.537908 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:37.537881 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw" Oct 30 18:30:37.541220 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:37.541198 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q" Oct 30 18:30:37.708273 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:37.708196 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be072889-51a1-4341-8559-6f445961c745-kube-api-access-scs8l\" (UniqueName: \"kubernetes.io/projected/be072889-51a1-4341-8559-6f445961c745-kube-api-access-scs8l\") pod \"be072889-51a1-4341-8559-6f445961c745\" (UID: \"be072889-51a1-4341-8559-6f445961c745\") " Oct 30 18:30:37.708273 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:37.708257 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5184124-242d-445e-9341-59d13045fda9-kube-api-access-nsjcw\" (UniqueName: \"kubernetes.io/projected/c5184124-242d-445e-9341-59d13045fda9-kube-api-access-nsjcw\") pod \"c5184124-242d-445e-9341-59d13045fda9\" (UID: \"c5184124-242d-445e-9341-59d13045fda9\") " Oct 30 18:30:37.708475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:37.708349 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5184124-242d-445e-9341-59d13045fda9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5184124-242d-445e-9341-59d13045fda9-maas-api-service-ca\") pod \"c5184124-242d-445e-9341-59d13045fda9\" (UID: \"c5184124-242d-445e-9341-59d13045fda9\") " Oct 30 18:30:37.708475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:37.708468 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be072889-51a1-4341-8559-6f445961c745-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be072889-51a1-4341-8559-6f445961c745-maas-api-service-ca\") pod \"be072889-51a1-4341-8559-6f445961c745\" (UID: \"be072889-51a1-4341-8559-6f445961c745\") " Oct 30 18:30:37.708760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:37.708729 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5184124-242d-445e-9341-59d13045fda9-maas-api-service-ca" pod "c5184124-242d-445e-9341-59d13045fda9" (UID: "c5184124-242d-445e-9341-59d13045fda9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:37.708874 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:37.708798 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be072889-51a1-4341-8559-6f445961c745-maas-api-service-ca" pod "be072889-51a1-4341-8559-6f445961c745" (UID: "be072889-51a1-4341-8559-6f445961c745"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:37.709015 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:37.708999 4603 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5184124-242d-445e-9341-59d13045fda9-maas-api-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:30:37.709060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:37.709018 4603 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be072889-51a1-4341-8559-6f445961c745-maas-api-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:30:37.710494 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:37.710473 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5184124-242d-445e-9341-59d13045fda9-kube-api-access-nsjcw" pod "c5184124-242d-445e-9341-59d13045fda9" (UID: "c5184124-242d-445e-9341-59d13045fda9"). InnerVolumeSpecName "kube-api-access-nsjcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:37.710618 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:37.710590 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be072889-51a1-4341-8559-6f445961c745-kube-api-access-scs8l" pod "be072889-51a1-4341-8559-6f445961c745" (UID: "be072889-51a1-4341-8559-6f445961c745"). InnerVolumeSpecName "kube-api-access-scs8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:37.810754 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:37.810721 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scs8l\" (UniqueName: \"kubernetes.io/projected/be072889-51a1-4341-8559-6f445961c745-kube-api-access-scs8l\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:30:37.810754 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:37.810748 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsjcw\" (UniqueName: \"kubernetes.io/projected/c5184124-242d-445e-9341-59d13045fda9-kube-api-access-nsjcw\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:30:38.138304 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:38.138275 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:38.317988 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:38.317951 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:38.474595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:38.463624 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw" event={"ID":"be072889-51a1-4341-8559-6f445961c745","Type":"ContainerDied","Data":"c1c987a99b7154bc5348eecf54b046e08a97c05d8130c6f8c0970fc2195a59c5"} Oct 30 18:30:38.475889 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:38.475854 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw" Oct 30 18:30:38.476042 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:38.475859 4603 scope.go:122] "RemoveContainer" containerID="eaa3c8cfc11b1ba9fa4658831962da277cbda19dd2274af5ea4fd0cf81f203a6" Oct 30 18:30:38.510526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:38.493606 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q" event={"ID":"c5184124-242d-445e-9341-59d13045fda9","Type":"ContainerDied","Data":"26875a1d1654c173114044a568f26e47db79d76a443d37dc2bb3b8f2d6dee495"} Oct 30 18:30:38.510526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:38.493723 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q" Oct 30 18:30:38.510526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:38.493859 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:38.782856 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:38.768894 4603 scope.go:122] "RemoveContainer" containerID="4904b7e53a1ae60bab32e566588a4d8dd5d1ab0b6913fc2f1fd0a4eca100f480" Oct 30 18:30:38.845649 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:38.845624 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw"] Oct 30 18:30:38.855793 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:38.855771 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77db82-29889750-xxhcw"] Oct 30 18:30:38.881619 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:38.881595 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q"] Oct 30 18:30:38.885719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:38.885697 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-v4r4q"] Oct 30 18:30:39.413246 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:39.413215 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9efc340c-4b4f-4a7f-ace1-0800c843361a" path="/var/lib/kubelet/pods/9efc340c-4b4f-4a7f-ace1-0800c843361a/volumes" Oct 30 18:30:39.413690 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:39.413673 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be072889-51a1-4341-8559-6f445961c745" path="/var/lib/kubelet/pods/be072889-51a1-4341-8559-6f445961c745/volumes" Oct 30 18:30:39.414086 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:39.414070 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5184124-242d-445e-9341-59d13045fda9" path="/var/lib/kubelet/pods/c5184124-242d-445e-9341-59d13045fda9/volumes" Oct 30 18:30:39.867746 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:39.867713 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:40.027958 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.027925 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw"] Oct 30 18:30:40.028701 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.028680 4603 cpu_manager.go:414] "containerMap: removing container" podUID="be072889-51a1-4341-8559-6f445961c745" containerName="cleanup" Oct 30 18:30:40.028701 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.028700 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be072889-51a1-4341-8559-6f445961c745" containerName="cleanup" Oct 30 18:30:40.028910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.028718 4603 cpu_manager.go:414] "containerMap: removing container" podUID="80914a10-e845-4294-9423-7d356e84d57b" containerName="cleanup" Oct 30 18:30:40.028910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.028727 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80914a10-e845-4294-9423-7d356e84d57b" containerName="cleanup" Oct 30 18:30:40.028910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.028740 4603 cpu_manager.go:414] "containerMap: removing container" podUID="9efc340c-4b4f-4a7f-ace1-0800c843361a" containerName="cleanup" Oct 30 18:30:40.028910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.028747 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9efc340c-4b4f-4a7f-ace1-0800c843361a" containerName="cleanup" Oct 30 18:30:40.028910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.028760 4603 cpu_manager.go:414] "containerMap: removing container" podUID="cf18bbde-1895-48b6-9ad1-7f6560660f8b" containerName="cleanup" Oct 30 18:30:40.028910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.028767 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf18bbde-1895-48b6-9ad1-7f6560660f8b" containerName="cleanup" Oct 30 18:30:40.028910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.028786 4603 cpu_manager.go:414] "containerMap: removing container" podUID="9efc340c-4b4f-4a7f-ace1-0800c843361a" containerName="cleanup" Oct 30 18:30:40.028910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.028793 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9efc340c-4b4f-4a7f-ace1-0800c843361a" containerName="cleanup" Oct 30 18:30:40.028910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.028813 4603 cpu_manager.go:414] "containerMap: removing container" podUID="be072889-51a1-4341-8559-6f445961c745" containerName="cleanup" Oct 30 18:30:40.028910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.028820 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be072889-51a1-4341-8559-6f445961c745" containerName="cleanup" Oct 30 18:30:40.028910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.028835 4603 cpu_manager.go:414] "containerMap: removing container" podUID="c5184124-242d-445e-9341-59d13045fda9" containerName="cleanup" Oct 30 18:30:40.028910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.028842 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5184124-242d-445e-9341-59d13045fda9" containerName="cleanup" Oct 30 18:30:40.028910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.028872 4603 cpu_manager.go:414] "containerMap: removing container" podUID="80914a10-e845-4294-9423-7d356e84d57b" containerName="cleanup" Oct 30 18:30:40.028910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.028880 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80914a10-e845-4294-9423-7d356e84d57b" containerName="cleanup" Oct 30 18:30:40.028910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.028906 4603 cpu_manager.go:414] "containerMap: removing container" podUID="cf18bbde-1895-48b6-9ad1-7f6560660f8b" containerName="cleanup" Oct 30 18:30:40.028910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.028913 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf18bbde-1895-48b6-9ad1-7f6560660f8b" containerName="cleanup" Oct 30 18:30:40.029716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.028935 4603 cpu_manager.go:414] "containerMap: removing container" podUID="c5184124-242d-445e-9341-59d13045fda9" containerName="cleanup" Oct 30 18:30:40.029716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.028944 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5184124-242d-445e-9341-59d13045fda9" containerName="cleanup" Oct 30 18:30:40.029716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.028954 4603 cpu_manager.go:414] "containerMap: removing container" podUID="c5184124-242d-445e-9341-59d13045fda9" containerName="cleanup" Oct 30 18:30:40.029716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.028960 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5184124-242d-445e-9341-59d13045fda9" containerName="cleanup" Oct 30 18:30:40.029716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.029141 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="9efc340c-4b4f-4a7f-ace1-0800c843361a" containerName="cleanup" Oct 30 18:30:40.029716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.029157 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="80914a10-e845-4294-9423-7d356e84d57b" containerName="cleanup" Oct 30 18:30:40.029716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.029168 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="be072889-51a1-4341-8559-6f445961c745" containerName="cleanup" Oct 30 18:30:40.029716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.029179 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5184124-242d-445e-9341-59d13045fda9" containerName="cleanup" Oct 30 18:30:40.029716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.029190 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="80914a10-e845-4294-9423-7d356e84d57b" containerName="cleanup" Oct 30 18:30:40.029716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.029201 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf18bbde-1895-48b6-9ad1-7f6560660f8b" containerName="cleanup" Oct 30 18:30:40.029716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.029210 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="be072889-51a1-4341-8559-6f445961c745" containerName="cleanup" Oct 30 18:30:40.029716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.029219 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf18bbde-1895-48b6-9ad1-7f6560660f8b" containerName="cleanup" Oct 30 18:30:40.029716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.029229 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5184124-242d-445e-9341-59d13045fda9" containerName="cleanup" Oct 30 18:30:40.029716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.029238 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5184124-242d-445e-9341-59d13045fda9" containerName="cleanup" Oct 30 18:30:40.029716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.029246 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="be072889-51a1-4341-8559-6f445961c745" containerName="cleanup" Oct 30 18:30:40.029716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.029254 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="80914a10-e845-4294-9423-7d356e84d57b" containerName="cleanup" Oct 30 18:30:40.029716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.029262 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="9efc340c-4b4f-4a7f-ace1-0800c843361a" containerName="cleanup" Oct 30 18:30:40.034997 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.034975 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw" Oct 30 18:30:40.061402 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.061370 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw"] Oct 30 18:30:40.062584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.062560 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:40.067886 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.067858 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-3b8bf3\"" Oct 30 18:30:40.068006 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.067903 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-3b8bf3-dockercfg-6k4dg\"" Oct 30 18:30:40.101675 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.101652 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j"] Oct 30 18:30:40.102299 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.102282 4603 cpu_manager.go:414] "containerMap: removing container" podUID="be072889-51a1-4341-8559-6f445961c745" containerName="cleanup" Oct 30 18:30:40.102299 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.102299 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be072889-51a1-4341-8559-6f445961c745" containerName="cleanup" Oct 30 18:30:40.102476 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.102313 4603 cpu_manager.go:414] "containerMap: removing container" podUID="cf18bbde-1895-48b6-9ad1-7f6560660f8b" containerName="cleanup" Oct 30 18:30:40.102476 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.102318 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf18bbde-1895-48b6-9ad1-7f6560660f8b" containerName="cleanup" Oct 30 18:30:40.102476 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.102333 4603 cpu_manager.go:414] "containerMap: removing container" podUID="9efc340c-4b4f-4a7f-ace1-0800c843361a" containerName="cleanup" Oct 30 18:30:40.102476 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.102337 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9efc340c-4b4f-4a7f-ace1-0800c843361a" containerName="cleanup" Oct 30 18:30:40.102476 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.102365 4603 cpu_manager.go:414] "containerMap: removing container" podUID="80914a10-e845-4294-9423-7d356e84d57b" containerName="cleanup" Oct 30 18:30:40.102476 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.102369 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80914a10-e845-4294-9423-7d356e84d57b" containerName="cleanup" Oct 30 18:30:40.102476 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.102466 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="9efc340c-4b4f-4a7f-ace1-0800c843361a" containerName="cleanup" Oct 30 18:30:40.102892 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.102481 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf18bbde-1895-48b6-9ad1-7f6560660f8b" containerName="cleanup" Oct 30 18:30:40.105275 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.105257 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j" Oct 30 18:30:40.202601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.193165 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e2e8d0b-1be6-4e07-aa98-db2472756ee5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw\" (UID: \"9e2e8d0b-1be6-4e07-aa98-db2472756ee5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw" Oct 30 18:30:40.202601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.193314 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crtx2\" (UniqueName: \"kubernetes.io/projected/9e2e8d0b-1be6-4e07-aa98-db2472756ee5-kube-api-access-crtx2\") pod \"payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw\" (UID: \"9e2e8d0b-1be6-4e07-aa98-db2472756ee5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw" Oct 30 18:30:40.299660 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.299621 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70342038-ca6c-4a23-980b-7b750af92367-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j\" (UID: \"70342038-ca6c-4a23-980b-7b750af92367\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j" Oct 30 18:30:40.299965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.299941 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e2e8d0b-1be6-4e07-aa98-db2472756ee5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw\" (UID: \"9e2e8d0b-1be6-4e07-aa98-db2472756ee5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw" Oct 30 18:30:40.300195 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.300179 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crtx2\" (UniqueName: \"kubernetes.io/projected/9e2e8d0b-1be6-4e07-aa98-db2472756ee5-kube-api-access-crtx2\") pod \"payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw\" (UID: \"9e2e8d0b-1be6-4e07-aa98-db2472756ee5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw" Oct 30 18:30:40.300404 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.300387 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hphf7\" (UniqueName: \"kubernetes.io/projected/70342038-ca6c-4a23-980b-7b750af92367-kube-api-access-hphf7\") pod \"payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j\" (UID: \"70342038-ca6c-4a23-980b-7b750af92367\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j" Oct 30 18:30:40.301588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.301563 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e2e8d0b-1be6-4e07-aa98-db2472756ee5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw\" (UID: \"9e2e8d0b-1be6-4e07-aa98-db2472756ee5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw" Oct 30 18:30:40.353983 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.353950 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j"] Oct 30 18:30:40.437974 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.437926 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hphf7\" (UniqueName: \"kubernetes.io/projected/70342038-ca6c-4a23-980b-7b750af92367-kube-api-access-hphf7\") pod \"payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j\" (UID: \"70342038-ca6c-4a23-980b-7b750af92367\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j" Oct 30 18:30:40.438748 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.438711 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70342038-ca6c-4a23-980b-7b750af92367-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j\" (UID: \"70342038-ca6c-4a23-980b-7b750af92367\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j" Oct 30 18:30:40.439947 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.439911 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70342038-ca6c-4a23-980b-7b750af92367-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j\" (UID: \"70342038-ca6c-4a23-980b-7b750af92367\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j" Oct 30 18:30:40.544615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.523944 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:40.544615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.544384 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crtx2\" (UniqueName: \"kubernetes.io/projected/9e2e8d0b-1be6-4e07-aa98-db2472756ee5-kube-api-access-crtx2\") pod \"payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw\" (UID: \"9e2e8d0b-1be6-4e07-aa98-db2472756ee5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw" Oct 30 18:30:40.563461 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.563421 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hphf7\" (UniqueName: \"kubernetes.io/projected/70342038-ca6c-4a23-980b-7b750af92367-kube-api-access-hphf7\") pod \"payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j\" (UID: \"70342038-ca6c-4a23-980b-7b750af92367\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j" Oct 30 18:30:40.687533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.683319 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j" Oct 30 18:30:40.703199 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:40.701374 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw" Oct 30 18:30:41.031848 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:30:41.031817 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70342038_ca6c_4a23_980b_7b750af92367.slice/crio-6ada50fc79e68b8149b35dd4d51671c9ea50aed6646cd1cded9a5476e482fe84 WatchSource:0}: Error finding container 6ada50fc79e68b8149b35dd4d51671c9ea50aed6646cd1cded9a5476e482fe84: Status 404 returned error can't find the container with id 6ada50fc79e68b8149b35dd4d51671c9ea50aed6646cd1cded9a5476e482fe84 Oct 30 18:30:41.031995 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:41.031977 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j"] Oct 30 18:30:41.050831 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:41.050810 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw"] Oct 30 18:30:41.051287 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:30:41.051263 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2e8d0b_1be6_4e07_aa98_db2472756ee5.slice/crio-c4476e4a5db72bd3e1e2207536328e0ff77ef649b01888adbb45aeeb9237dce4 WatchSource:0}: Error finding container c4476e4a5db72bd3e1e2207536328e0ff77ef649b01888adbb45aeeb9237dce4: Status 404 returned error can't find the container with id c4476e4a5db72bd3e1e2207536328e0ff77ef649b01888adbb45aeeb9237dce4 Oct 30 18:30:41.824566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:41.824519 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw" event={"ID":"9e2e8d0b-1be6-4e07-aa98-db2472756ee5","Type":"ContainerStarted","Data":"b34a6cf8ba2569641d9186e13fc4c33879c0f5616f0df2180949f1ca697b75fe"} Oct 30 18:30:41.824996 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:41.824589 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw" Oct 30 18:30:41.824996 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:41.824612 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw" event={"ID":"9e2e8d0b-1be6-4e07-aa98-db2472756ee5","Type":"ContainerStarted","Data":"c4476e4a5db72bd3e1e2207536328e0ff77ef649b01888adbb45aeeb9237dce4"} Oct 30 18:30:41.826272 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:41.826244 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j" event={"ID":"70342038-ca6c-4a23-980b-7b750af92367","Type":"ContainerStarted","Data":"9072d87870a27eb2725933e9f9637ec2be47b30e1812b9dbfab97a3778a0e708"} Oct 30 18:30:41.826399 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:41.826293 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j" Oct 30 18:30:41.826399 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:41.826309 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j" event={"ID":"70342038-ca6c-4a23-980b-7b750af92367","Type":"ContainerStarted","Data":"6ada50fc79e68b8149b35dd4d51671c9ea50aed6646cd1cded9a5476e482fe84"} Oct 30 18:30:41.866516 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:41.866459 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw" podStartSLOduration=2.8664463209999997 podStartE2EDuration="2.866446321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:41.863900309 +0000 UTC m=+2302.579470183" watchObservedRunningTime="2026-10-30 18:30:41.866446321 +0000 UTC m=+2302.582016201" Oct 30 18:30:41.904483 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:41.904428 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j" podStartSLOduration=1.9044114109999999 podStartE2EDuration="1.904411411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:41.903571806 +0000 UTC m=+2302.619141680" watchObservedRunningTime="2026-10-30 18:30:41.904411411 +0000 UTC m=+2302.619981284" Oct 30 18:30:42.052667 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:42.052635 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:43.753015 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:43.752969 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:45.432170 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:45.432118 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:47.081465 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:47.081431 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:47.313895 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:47.313865 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc" Oct 30 18:30:47.314080 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:47.313947 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp" Oct 30 18:30:48.888099 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:48.888059 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:51.176673 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:51.172845 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:51.933358 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:30:51.933322 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:30:52.847274 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:52.842673 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:52.886525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:52.883969 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw" Oct 30 18:30:52.889221 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:52.888689 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j" Oct 30 18:30:53.022005 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:53.021971 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:54.407275 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:54.407237 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:54.586599 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:54.586326 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:56.193313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:56.193274 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:57.949584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:57.949548 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:59.700229 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:30:59.700194 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:01.255310 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:01.255271 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:01.600537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:01.588793 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:03.320327 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.320296 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46"] Oct 30 18:31:03.324992 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.324964 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:03.342333 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.342307 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-a2a10a\"/\"default-dockercfg-ksgnp\"" Oct 30 18:31:03.353213 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.353176 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-a2a10a\"/\"auth-test-model-a2a10a-kserve-self-signed-certs\"" Oct 30 18:31:03.353213 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.353197 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-a2a10a\"/\"openshift-service-ca.crt\"" Oct 30 18:31:03.353381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.353200 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-a2a10a\"/\"kube-root-ca.crt\"" Oct 30 18:31:03.357244 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.357223 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46"] Oct 30 18:31:03.444252 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.444219 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:03.492883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.492859 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-model-cache\") pod \"auth-test-model-a2a10a-kserve-867f45cf6f-sqh46\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:03.493038 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.492889 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a35d239-64fb-4752-91e9-10f61d87713f-tls-certs\") pod \"auth-test-model-a2a10a-kserve-867f45cf6f-sqh46\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:03.493038 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.492933 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-home\") pod \"auth-test-model-a2a10a-kserve-867f45cf6f-sqh46\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:03.493137 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.493057 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-dshm\") pod \"auth-test-model-a2a10a-kserve-867f45cf6f-sqh46\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:03.493137 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.493120 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-tmp-dir\") pod \"auth-test-model-a2a10a-kserve-867f45cf6f-sqh46\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:03.493254 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.493225 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjtpf\" (UniqueName: \"kubernetes.io/projected/2a35d239-64fb-4752-91e9-10f61d87713f-kube-api-access-wjtpf\") pod \"auth-test-model-a2a10a-kserve-867f45cf6f-sqh46\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:03.615008 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.595241 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-dshm\") pod \"auth-test-model-a2a10a-kserve-867f45cf6f-sqh46\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:03.615466 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.615435 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-tmp-dir\") pod \"auth-test-model-a2a10a-kserve-867f45cf6f-sqh46\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:03.616348 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.616323 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjtpf\" (UniqueName: \"kubernetes.io/projected/2a35d239-64fb-4752-91e9-10f61d87713f-kube-api-access-wjtpf\") pod \"auth-test-model-a2a10a-kserve-867f45cf6f-sqh46\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:03.617395 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.617370 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-model-cache\") pod \"auth-test-model-a2a10a-kserve-867f45cf6f-sqh46\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:03.617937 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.617909 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a35d239-64fb-4752-91e9-10f61d87713f-tls-certs\") pod \"auth-test-model-a2a10a-kserve-867f45cf6f-sqh46\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:03.618955 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.618933 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-home\") pod \"auth-test-model-a2a10a-kserve-867f45cf6f-sqh46\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:03.619493 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.616201 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-tmp-dir\") pod \"auth-test-model-a2a10a-kserve-867f45cf6f-sqh46\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:03.620057 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.619989 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-home\") pod \"auth-test-model-a2a10a-kserve-867f45cf6f-sqh46\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:03.620204 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.617838 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-model-cache\") pod \"auth-test-model-a2a10a-kserve-867f45cf6f-sqh46\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:03.648398 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.643902 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-dshm\") pod \"auth-test-model-a2a10a-kserve-867f45cf6f-sqh46\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:03.648398 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.645086 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a35d239-64fb-4752-91e9-10f61d87713f-tls-certs\") pod \"auth-test-model-a2a10a-kserve-867f45cf6f-sqh46\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:03.669855 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.649323 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjtpf\" (UniqueName: \"kubernetes.io/projected/2a35d239-64fb-4752-91e9-10f61d87713f-kube-api-access-wjtpf\") pod \"auth-test-model-a2a10a-kserve-867f45cf6f-sqh46\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:03.945013 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.944911 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:03.945686 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:03.945669 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:04.383166 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:04.383143 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46"] Oct 30 18:31:04.384218 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:31:04.384195 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a35d239_64fb_4752_91e9_10f61d87713f.slice/crio-e41e427be3eded20f438b2e64cb83576e5f5914094ee1a3be2ff95f468e54b17 WatchSource:0}: Error finding container e41e427be3eded20f438b2e64cb83576e5f5914094ee1a3be2ff95f468e54b17: Status 404 returned error can't find the container with id e41e427be3eded20f438b2e64cb83576e5f5914094ee1a3be2ff95f468e54b17 Oct 30 18:31:05.119171 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:05.119139 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" event={"ID":"2a35d239-64fb-4752-91e9-10f61d87713f","Type":"ContainerStarted","Data":"2c1c1f39a54cb8aebcf2dbf7f6f64c5d2975cfec95267ab955ec321eb6f6cddd"} Oct 30 18:31:05.119171 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:05.119177 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" event={"ID":"2a35d239-64fb-4752-91e9-10f61d87713f","Type":"ContainerStarted","Data":"e41e427be3eded20f438b2e64cb83576e5f5914094ee1a3be2ff95f468e54b17"} Oct 30 18:31:05.172167 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:05.172116 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podStartSLOduration=2.172104074 podStartE2EDuration="2.172104074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:05.168115862 +0000 UTC m=+2325.883685735" watchObservedRunningTime="2026-10-30 18:31:05.172104074 +0000 UTC m=+2325.887673982" Oct 30 18:31:05.738584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:05.738552 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:05.936198 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:05.936125 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:05.957155 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:05.957127 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:07.638409 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:07.638370 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:09.325271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:09.325226 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:10.964838 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:10.964778 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:12.648403 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:12.648364 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:13.945852 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:13.945816 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:13.957995 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:13.957966 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:14.216195 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:14.216117 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:14.341414 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:14.341385 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:14.629116 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:14.629077 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:14.779693 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:14.779648 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:16.168140 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:16.168096 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:17.607141 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:17.607107 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:17.882946 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:17.882865 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:18.205173 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:18.205075 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:20.261130 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:20.261095 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:21.932242 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:31:21.932206 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:31:21.944623 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:21.944594 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:23.646549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:23.639595 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:25.505582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:25.505546 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:27.307022 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:27.306993 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:28.528779 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:28.528739 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:28.963993 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:28.963908 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:29.380205 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:29.380176 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:30.818717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:30.818680 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:33.182415 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:33.182373 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:34.861923 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:34.861886 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:36.631078 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:36.631045 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:37.664036 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:37.663998 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:31:38.468625 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:38.468586 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:39.621327 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:39.621292 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:40.994557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:40.985314 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:41.664528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:41.634594 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:41.773530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:41.768141 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:31:41.830869 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:41.830802 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:41.986591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:41.982606 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:31:42.090672 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:42.090601 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:42.213543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:42.211661 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:42.249594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:42.247459 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:31:42.249594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:42.247547 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" podUID="d5567046-aa24-421a-8339-fdd9cf56e62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:42.312634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:42.312589 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:42.361543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:42.359604 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:42.384527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:42.378593 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:42.673534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:42.642883 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:42.687537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:42.674212 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:31:42.785817 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:42.785775 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:31:42.843544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:42.832189 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:31:42.918112 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:42.917602 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:42.918112 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:42.917922 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:31:42.983559 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:42.983439 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:43.105126 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:43.103083 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:31:43.122019 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:43.106778 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:31:43.124379 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:43.123275 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:31:43.266613 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:43.259346 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:43.304557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:43.285120 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:43.373138 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:43.346174 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:31:43.373762 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:43.365597 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:43.375035 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:43.375012 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:31:43.406174 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:43.382589 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:43.734650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:43.734597 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:43.879620 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:43.850583 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:43.948875 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:43.944367 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:31:43.997286 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:43.997196 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:31:44.023058 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.023032 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:31:44.055365 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.005907 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:44.164444 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:43.999606 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:44.173729 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.108405 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:44.315298 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.281999 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:44.384432 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.351981 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:44.385088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.382600 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:31:44.412992 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.412952 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:44.440639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.440596 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:31:44.557438 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.548192 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq"] Oct 30 18:31:44.557438 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.555779 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:31:44.622585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.613142 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-b00d69\"/\"default-dockercfg-pnc8t\"" Oct 30 18:31:44.622585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.613728 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-b00d69\"/\"openshift-service-ca.crt\"" Oct 30 18:31:44.622585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.613905 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-b00d69\"/\"auth-test-model-b00d69-kserve-self-signed-certs\"" Oct 30 18:31:44.622585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.619678 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-b00d69\"/\"kube-root-ca.crt\"" Oct 30 18:31:44.672277 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.656905 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:44.745930 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.745874 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:44.839872 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.839715 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq"] Oct 30 18:31:44.888590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.887827 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:44.957532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.941985 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc4f5be5-e07f-4960-ace5-4023198470be-tls-certs\") pod \"auth-test-model-b00d69-kserve-5ff49f965c-8l5zq\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:31:44.982758 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.982717 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-dshm\") pod \"auth-test-model-b00d69-kserve-5ff49f965c-8l5zq\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:31:44.983033 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.983009 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-tmp-dir\") pod \"auth-test-model-b00d69-kserve-5ff49f965c-8l5zq\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:31:44.983219 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.983199 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwvtz\" (UniqueName: \"kubernetes.io/projected/cc4f5be5-e07f-4960-ace5-4023198470be-kube-api-access-lwvtz\") pod \"auth-test-model-b00d69-kserve-5ff49f965c-8l5zq\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:31:44.983439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.983422 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-home\") pod \"auth-test-model-b00d69-kserve-5ff49f965c-8l5zq\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:31:44.997585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.983978 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-model-cache\") pod \"auth-test-model-b00d69-kserve-5ff49f965c-8l5zq\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:31:45.023528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.946617 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:45.024129 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:44.973262 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/health\": context deadline exceeded" Oct 30 18:31:45.101823 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:45.101779 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-dshm\") pod \"auth-test-model-b00d69-kserve-5ff49f965c-8l5zq\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:31:45.133963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:45.133933 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-tmp-dir\") pod \"auth-test-model-b00d69-kserve-5ff49f965c-8l5zq\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:31:45.134239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:45.134220 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwvtz\" (UniqueName: \"kubernetes.io/projected/cc4f5be5-e07f-4960-ace5-4023198470be-kube-api-access-lwvtz\") pod \"auth-test-model-b00d69-kserve-5ff49f965c-8l5zq\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:31:45.134365 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:45.134353 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-home\") pod \"auth-test-model-b00d69-kserve-5ff49f965c-8l5zq\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:31:45.156726 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:45.134780 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-model-cache\") pod \"auth-test-model-b00d69-kserve-5ff49f965c-8l5zq\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:31:45.158286 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:45.158259 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc4f5be5-e07f-4960-ace5-4023198470be-tls-certs\") pod \"auth-test-model-b00d69-kserve-5ff49f965c-8l5zq\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:31:45.175411 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:45.147059 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-home\") pod \"auth-test-model-b00d69-kserve-5ff49f965c-8l5zq\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:31:45.190590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:45.136412 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-model-cache\") pod \"auth-test-model-b00d69-kserve-5ff49f965c-8l5zq\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:31:45.190814 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:45.177586 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:31:45.191251 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:45.178914 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:45.214909 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:45.199552 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-tmp-dir\") pod \"auth-test-model-b00d69-kserve-5ff49f965c-8l5zq\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:31:45.229799 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:45.229765 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-dshm\") pod \"auth-test-model-b00d69-kserve-5ff49f965c-8l5zq\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:31:45.375483 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:45.234775 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc4f5be5-e07f-4960-ace5-4023198470be-tls-certs\") pod \"auth-test-model-b00d69-kserve-5ff49f965c-8l5zq\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:31:45.375483 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:45.257310 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwvtz\" (UniqueName: \"kubernetes.io/projected/cc4f5be5-e07f-4960-ace5-4023198470be-kube-api-access-lwvtz\") pod \"auth-test-model-b00d69-kserve-5ff49f965c-8l5zq\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:31:45.384650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:45.384606 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": context deadline exceeded" Oct 30 18:31:45.385897 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:45.385862 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:45.415578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:45.405819 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:31:45.421693 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:45.421650 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:45.520751 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:45.520707 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:45.615644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:45.607772 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:45.800860 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:45.751168 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:31:45.903578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:45.902279 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:31:46.030844 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:46.030796 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:46.194727 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:46.194630 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:46.218354 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:46.218315 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:46.416828 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:46.416788 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": context deadline exceeded" Oct 30 18:31:46.417434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:46.417403 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:31:46.435647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:46.435601 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:46.623347 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:46.620182 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:46.819539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:46.760459 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:46.985579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:46.957722 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:47.076528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:47.066653 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:31:47.202528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:47.182930 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:47.219607 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:47.203661 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:47.225340 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:47.225302 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:31:47.398526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:47.382751 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded" Oct 30 18:31:47.426457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:47.426414 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:47.427025 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:47.426993 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:47.519206 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:47.517001 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:47.631096 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:47.631055 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:47.762887 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:47.762835 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:31:48.003583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:47.999760 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq"] Oct 30 18:31:48.098544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:48.098367 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:31:48.099879 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:48.099828 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:48.132969 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:48.132909 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:48.355243 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:48.353026 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:48.373953 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:48.373911 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:48.426981 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:48.426939 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:31:48.427625 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:48.427594 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 30 18:31:48.428594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:48.428566 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:48.622616 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:48.611229 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Oct 30 18:31:48.622616 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:48.612750 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:31:48.635728 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:48.634006 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:48.822534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:48.778962 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:48.955098 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:48.954990 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" event={"ID":"cc4f5be5-e07f-4960-ace5-4023198470be","Type":"ContainerStarted","Data":"03b7a6b6f3408300e0c4d33bb7af8f7ca961dd94ebe47025645d28ffa9dcbdb4"} Oct 30 18:31:49.039529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:49.017364 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:31:49.039529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:49.017450 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" podUID="4d6cd466-9f15-476d-9f04-6d03176f025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:49.102007 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:49.101965 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:49.102700 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:49.102302 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:31:49.185643 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:49.185588 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" podUID="b664f4ca-736e-4cb1-991d-985d1e6aff34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:49.225021 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:49.224925 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:31:49.276528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:49.274394 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:31:49.276528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:49.274462 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" podUID="1d187b51-f5c6-4a63-8300-2f01184a15d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:49.276898 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:49.276871 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:31:49.294560 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:49.277025 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" podUID="d4a3adf6-fb23-4334-8bab-f800b4e8e089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:49.366885 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:49.364717 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:49.385643 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:49.385601 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:31:49.464672 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:49.464626 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:49.465300 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:49.465265 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:31:49.657530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:49.630144 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:31:49.657530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:49.643730 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:49.816575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:49.805231 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:49.914764 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:49.888697 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:50.110101 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:50.110034 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:50.232740 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:50.232636 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:50.308847 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:50.308605 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:50.373189 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:50.372602 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:50.530597 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:50.411574 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:50.568681 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:50.490732 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:50.568681 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:50.522288 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:50.659546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:50.656272 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:50.659546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:50.658821 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:31:50.928601 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:50.926141 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:31:51.128270 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:51.128216 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:51.323765 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:51.305198 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:31:51.433529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:51.410206 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:31:51.567037 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:51.566990 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:51.572528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:51.570381 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:31:51.576935 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:51.573015 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": context deadline exceeded" Oct 30 18:31:51.748526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:51.745249 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:31:51.778529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:51.776899 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:52.013345 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:52.013234 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:31:52.169535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:52.168860 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:52.324538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:52.312454 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:52.506529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:52.505761 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:31:52.589828 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:52.589720 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:52.612694 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:52.612642 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:52.709532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:52.702664 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:31:52.748657 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:52.745798 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:52.835529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:52.785571 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:52.856923 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:52.785771 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:52.856923 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:31:52.855808 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:31:53.167681 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:53.016769 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:53.167681 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:53.071625 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:31:53.167681 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:53.071710 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" podUID="d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:53.605202 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:53.605146 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:53.658527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:53.605864 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" podUID="1112c14a-18f4-46de-ba0c-8e28f19ef80d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded" Oct 30 18:31:53.791014 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:53.790958 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:53.824727 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:53.824683 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:53.825894 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:53.825863 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": context deadline exceeded" Oct 30 18:31:53.875740 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:53.846248 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:31:53.875740 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:53.659223 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:31:53.916527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:53.909223 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:53.917161 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:53.917116 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" podUID="f0702ace-9c54-4015-add0-e25ee9feaf40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded" Oct 30 18:31:54.031496 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:53.930571 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:31:54.032373 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:53.972923 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:31:54.032585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:54.032556 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" podUID="58e2469b-dc0d-40ab-bd74-a47c4a9e438b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:54.142646 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:54.103524 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:31:54.436612 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:54.425396 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:54.635795 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:54.635741 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:54.645627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:54.645544 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:31:54.708861 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:54.708772 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:31:54.826077 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:54.826028 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:31:54.844655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:54.844617 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:54.921776 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:54.921726 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:54.922372 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:54.922341 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:31:54.938681 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:54.938646 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:55.008810 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:54.997818 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:31:55.040612 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:55.040583 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:31:55.067754 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:55.060956 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:31:55.125629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:55.125593 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:55.217781 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:55.217752 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:31:55.326644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:55.326612 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:31:55.388006 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:55.387918 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:31:55.468530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:55.467660 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:31:56.059982 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:56.059955 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6_2f5e20ee-f725-4245-be00-4df9759c4a50/payload-processing/0.log" Oct 30 18:31:56.158569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:56.158530 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" event={"ID":"cc4f5be5-e07f-4960-ace5-4023198470be","Type":"ContainerStarted","Data":"91951558205df012a4ce56c76f19d13d9a8fe05fcad84b323706e3529a340bb7"} Oct 30 18:31:56.238620 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:56.238554 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podStartSLOduration=12.23849372 podStartE2EDuration="12.23849372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:56.212918697 +0000 UTC m=+2376.928488595" watchObservedRunningTime="2026-10-30 18:31:56.23849372 +0000 UTC m=+2376.954063594" Oct 30 18:31:56.410709 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:56.410616 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:31:56.640411 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:56.640379 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:56.822091 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:56.822064 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9_b79b5853-8745-436a-aeff-aada86fb4853/payload-processing/0.log" Oct 30 18:31:57.022839 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:57.022809 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9_b79b5853-8745-436a-aeff-aada86fb4853/payload-processing/0.log" Oct 30 18:31:58.388540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:31:58.369212 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:00.069001 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:00.068965 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:01.824423 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:01.824380 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:03.371130 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:03.371093 4603 generic.go:356] "Generic (PLEG): container finished" podID="67142e99-ffa0-4b21-8f93-735d5663c87e" containerID="8aa33ec0ada9b6affb208856da6a1cfbbfa39a7f97414d9d8cc44e1bd2b210f6" exitCode=0 Oct 30 18:32:03.371669 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:03.371166 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" event={"ID":"67142e99-ffa0-4b21-8f93-735d5663c87e","Type":"ContainerDied","Data":"8aa33ec0ada9b6affb208856da6a1cfbbfa39a7f97414d9d8cc44e1bd2b210f6"} Oct 30 18:32:03.371669 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:03.371613 4603 scope.go:122] "RemoveContainer" containerID="8aa33ec0ada9b6affb208856da6a1cfbbfa39a7f97414d9d8cc44e1bd2b210f6" Oct 30 18:32:03.373106 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:03.373088 4603 generic.go:356] "Generic (PLEG): container finished" podID="73366d41-4910-43f0-aad8-2542401ec0f6" containerID="d95c310ab51db3b9a868be677d78cc335b2b18e6ac980e855881ec4e704755d7" exitCode=0 Oct 30 18:32:03.373192 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:03.373107 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" event={"ID":"73366d41-4910-43f0-aad8-2542401ec0f6","Type":"ContainerDied","Data":"d95c310ab51db3b9a868be677d78cc335b2b18e6ac980e855881ec4e704755d7"} Oct 30 18:32:03.373455 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:03.373442 4603 scope.go:122] "RemoveContainer" containerID="d95c310ab51db3b9a868be677d78cc335b2b18e6ac980e855881ec4e704755d7" Oct 30 18:32:03.373554 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:03.373495 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:03.373626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:03.373605 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:03.373662 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:03.373647 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:03.376357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:03.376096 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:03.376357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:03.376275 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:03.377534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:03.376728 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:03.613528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:03.606394 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:04.383004 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:04.382969 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" event={"ID":"73366d41-4910-43f0-aad8-2542401ec0f6","Type":"ContainerStarted","Data":"0088043c96740ee34b18139e21b425d3d572db30d303cfb0003cc5fb9539fba7"} Oct 30 18:32:04.385088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:04.385064 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" event={"ID":"67142e99-ffa0-4b21-8f93-735d5663c87e","Type":"ContainerStarted","Data":"c2e092dbe8ec0eeb04e8b29e3f37c0d574a81c98465b9c1d6e570fc363785e83"} Oct 30 18:32:05.231529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:05.227082 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:32:05.243819 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:05.241997 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 18:32:05.243819 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:05.242068 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" podUID="67142e99-ffa0-4b21-8f93-735d5663c87e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 18:32:05.261562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:05.261256 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:05.285668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:05.285640 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:32:05.286609 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:05.286578 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 18:32:05.286660 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:05.286628 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" podUID="73366d41-4910-43f0-aad8-2542401ec0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 18:32:05.571677 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:05.569452 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:32:05.576847 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:05.576825 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:32:06.227529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:06.227116 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 18:32:06.227529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:06.227173 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" podUID="67142e99-ffa0-4b21-8f93-735d5663c87e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 18:32:06.286116 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:06.286082 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 18:32:06.286283 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:06.286131 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" podUID="73366d41-4910-43f0-aad8-2542401ec0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 18:32:06.308529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:06.306594 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:32:06.456380 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:06.456350 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:32:06.546528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:06.545616 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:32:06.964110 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:06.964012 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:07.240738 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:07.240644 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 18:32:07.240909 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:07.240738 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" podUID="67142e99-ffa0-4b21-8f93-735d5663c87e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 18:32:07.285590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:07.285562 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 18:32:07.285760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:07.285619 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" podUID="73366d41-4910-43f0-aad8-2542401ec0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 18:32:08.225284 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:08.224825 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 18:32:08.225284 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:08.224899 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" podUID="67142e99-ffa0-4b21-8f93-735d5663c87e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 18:32:08.287142 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:08.287108 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 18:32:08.287310 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:08.287166 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" podUID="73366d41-4910-43f0-aad8-2542401ec0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 18:32:09.234530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:09.224804 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 18:32:09.234530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:09.224862 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" podUID="67142e99-ffa0-4b21-8f93-735d5663c87e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 18:32:09.286126 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:09.286097 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 18:32:09.286314 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:09.286152 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" podUID="73366d41-4910-43f0-aad8-2542401ec0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 18:32:09.566529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:09.565431 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:10.223768 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:10.223727 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 18:32:10.223971 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:10.223796 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" podUID="67142e99-ffa0-4b21-8f93-735d5663c87e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 18:32:10.286106 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:10.286071 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 18:32:10.286559 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:10.286125 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" podUID="73366d41-4910-43f0-aad8-2542401ec0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 18:32:11.229870 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:11.229839 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:32:11.230945 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:11.230923 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:32:11.252177 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:11.252148 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:11.346172 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:11.346131 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:32:11.372553 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:11.367744 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:32:12.845686 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:12.845651 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:14.458650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:14.458615 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:14.636858 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:14.636779 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:16.169218 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:16.169166 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:17.649099 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:17.649062 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:17.863595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:17.863555 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:18.324961 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:18.324931 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:19.669093 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:19.669056 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:19.821354 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:19.821320 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:20.701291 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:20.701264 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 18:32:20.996228 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:20.996148 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 18:32:21.438836 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:21.438804 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:21.930941 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:32:21.930904 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:32:23.146108 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:23.146061 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:24.700901 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:24.700858 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:26.417883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:26.417816 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:28.030142 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:28.030105 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:29.624428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:29.624392 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:31.238660 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:31.238627 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:32.957416 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:32.957380 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:35.153067 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:35.153032 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:36.830660 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:36.830609 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:38.546367 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:38.546327 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:39.997733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:39.997691 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:40.182406 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:40.182370 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:41.926774 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:41.926742 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:43.559309 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:43.559272 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:45.151131 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:45.151095 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:46.709728 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:46.709697 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:48.294695 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:48.294664 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:50.052808 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:50.052770 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:51.645551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:51.645516 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:51.933410 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:32:51.933330 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:32:53.121625 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:53.121591 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:53.524450 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:53.524120 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:53.976661 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:53.976463 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:54.060000 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:54.059967 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6"] Oct 30 18:32:54.060343 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:54.060316 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw"] Oct 30 18:32:54.060588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:54.060563 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6" podUID="2f5e20ee-f725-4245-be00-4df9759c4a50" containerName="payload-processing" containerID="cri-o://ae286b566c5394f52c893b33cd8af0c119fed0974fc0cabab24a626e2962494b" gracePeriod=30 Oct 30 18:32:54.061339 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:54.060980 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw" podUID="803ab509-6175-4535-a6b5-29bf877e9d72" containerName="payload-pre-processing" containerID="cri-o://48c8f9a8d6282d8d3da637130d40df86968d2eacddf7738656db68f36da975a2" gracePeriod=30 Oct 30 18:32:54.191722 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:54.191410 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:54.640568 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:54.640528 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:56.097665 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:56.086605 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:57.756753 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:57.756672 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:58.449173 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:58.449129 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw" podUID="803ab509-6175-4535-a6b5-29bf877e9d72" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.138:9004: connect: connection refused" Oct 30 18:32:59.467223 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:59.467186 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6" podUID="2f5e20ee-f725-4245-be00-4df9759c4a50" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.139:9004: connect: connection refused" Oct 30 18:32:59.470865 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:59.470837 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:59.679205 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:32:59.679167 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:00.403954 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:00.403918 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6" Oct 30 18:33:00.426148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:00.426119 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw" Oct 30 18:33:00.502446 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:00.502417 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/803ab509-6175-4535-a6b5-29bf877e9d72-kube-api-access-8jv6s\" (UniqueName: \"kubernetes.io/projected/803ab509-6175-4535-a6b5-29bf877e9d72-kube-api-access-8jv6s\") pod \"803ab509-6175-4535-a6b5-29bf877e9d72\" (UID: \"803ab509-6175-4535-a6b5-29bf877e9d72\") " Oct 30 18:33:00.502960 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:00.502476 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f5e20ee-f725-4245-be00-4df9759c4a50-kube-api-access-9rb8f\" (UniqueName: \"kubernetes.io/projected/2f5e20ee-f725-4245-be00-4df9759c4a50-kube-api-access-9rb8f\") pod \"2f5e20ee-f725-4245-be00-4df9759c4a50\" (UID: \"2f5e20ee-f725-4245-be00-4df9759c4a50\") " Oct 30 18:33:00.502960 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:00.502638 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f5e20ee-f725-4245-be00-4df9759c4a50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5e20ee-f725-4245-be00-4df9759c4a50-plugins-config-volume\") pod \"2f5e20ee-f725-4245-be00-4df9759c4a50\" (UID: \"2f5e20ee-f725-4245-be00-4df9759c4a50\") " Oct 30 18:33:00.502960 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:00.502896 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/803ab509-6175-4535-a6b5-29bf877e9d72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803ab509-6175-4535-a6b5-29bf877e9d72-plugins-config-volume\") pod \"803ab509-6175-4535-a6b5-29bf877e9d72\" (UID: \"803ab509-6175-4535-a6b5-29bf877e9d72\") " Oct 30 18:33:00.503279 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:00.503253 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5e20ee-f725-4245-be00-4df9759c4a50-plugins-config-volume" pod "2f5e20ee-f725-4245-be00-4df9759c4a50" (UID: "2f5e20ee-f725-4245-be00-4df9759c4a50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:00.503395 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:00.503372 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/803ab509-6175-4535-a6b5-29bf877e9d72-plugins-config-volume" pod "803ab509-6175-4535-a6b5-29bf877e9d72" (UID: "803ab509-6175-4535-a6b5-29bf877e9d72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:00.503828 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:00.503727 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5e20ee-f725-4245-be00-4df9759c4a50-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:00.503941 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:00.503869 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803ab509-6175-4535-a6b5-29bf877e9d72-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:00.505565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:00.505543 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803ab509-6175-4535-a6b5-29bf877e9d72-kube-api-access-8jv6s" pod "803ab509-6175-4535-a6b5-29bf877e9d72" (UID: "803ab509-6175-4535-a6b5-29bf877e9d72"). InnerVolumeSpecName "kube-api-access-8jv6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:00.506135 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:00.506102 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5e20ee-f725-4245-be00-4df9759c4a50-kube-api-access-9rb8f" pod "2f5e20ee-f725-4245-be00-4df9759c4a50" (UID: "2f5e20ee-f725-4245-be00-4df9759c4a50"). InnerVolumeSpecName "kube-api-access-9rb8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:00.604907 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:00.604879 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jv6s\" (UniqueName: \"kubernetes.io/projected/803ab509-6175-4535-a6b5-29bf877e9d72-kube-api-access-8jv6s\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:00.604907 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:00.604903 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rb8f\" (UniqueName: \"kubernetes.io/projected/2f5e20ee-f725-4245-be00-4df9759c4a50-kube-api-access-9rb8f\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:01.158919 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:01.158885 4603 generic.go:356] "Generic (PLEG): container finished" podID="2f5e20ee-f725-4245-be00-4df9759c4a50" containerID="ae286b566c5394f52c893b33cd8af0c119fed0974fc0cabab24a626e2962494b" exitCode=0 Oct 30 18:33:01.159087 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:01.158935 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6" Oct 30 18:33:01.159087 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:01.158966 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6" event={"ID":"2f5e20ee-f725-4245-be00-4df9759c4a50","Type":"ContainerDied","Data":"ae286b566c5394f52c893b33cd8af0c119fed0974fc0cabab24a626e2962494b"} Oct 30 18:33:01.159087 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:01.159075 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6" event={"ID":"2f5e20ee-f725-4245-be00-4df9759c4a50","Type":"ContainerDied","Data":"2993b9601ef5049a1fc2abc95587b6b5c98d12dac852a8d2a8165db99125e12b"} Oct 30 18:33:01.159250 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:01.159102 4603 scope.go:122] "RemoveContainer" containerID="ae286b566c5394f52c893b33cd8af0c119fed0974fc0cabab24a626e2962494b" Oct 30 18:33:01.160614 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:01.160582 4603 generic.go:356] "Generic (PLEG): container finished" podID="803ab509-6175-4535-a6b5-29bf877e9d72" containerID="48c8f9a8d6282d8d3da637130d40df86968d2eacddf7738656db68f36da975a2" exitCode=0 Oct 30 18:33:01.160749 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:01.160630 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw" event={"ID":"803ab509-6175-4535-a6b5-29bf877e9d72","Type":"ContainerDied","Data":"48c8f9a8d6282d8d3da637130d40df86968d2eacddf7738656db68f36da975a2"} Oct 30 18:33:01.160749 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:01.160681 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw" event={"ID":"803ab509-6175-4535-a6b5-29bf877e9d72","Type":"ContainerDied","Data":"4ce4137671f96c56b9a4f5639da4b2f5a8c1b925230e865c432c0e157c00536c"} Oct 30 18:33:01.160749 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:01.160636 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw" Oct 30 18:33:01.171953 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:01.171938 4603 scope.go:122] "RemoveContainer" containerID="ae286b566c5394f52c893b33cd8af0c119fed0974fc0cabab24a626e2962494b" Oct 30 18:33:01.172243 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:33:01.172218 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae286b566c5394f52c893b33cd8af0c119fed0974fc0cabab24a626e2962494b\": container with ID starting with ae286b566c5394f52c893b33cd8af0c119fed0974fc0cabab24a626e2962494b not found: ID does not exist" containerID="ae286b566c5394f52c893b33cd8af0c119fed0974fc0cabab24a626e2962494b" Oct 30 18:33:01.172320 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:01.172242 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae286b566c5394f52c893b33cd8af0c119fed0974fc0cabab24a626e2962494b"} err="failed to get container status \"ae286b566c5394f52c893b33cd8af0c119fed0974fc0cabab24a626e2962494b\": rpc error: code = NotFound desc = could not find container \"ae286b566c5394f52c893b33cd8af0c119fed0974fc0cabab24a626e2962494b\": container with ID starting with ae286b566c5394f52c893b33cd8af0c119fed0974fc0cabab24a626e2962494b not found: ID does not exist" Oct 30 18:33:01.172320 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:01.172260 4603 scope.go:122] "RemoveContainer" containerID="48c8f9a8d6282d8d3da637130d40df86968d2eacddf7738656db68f36da975a2" Oct 30 18:33:01.181785 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:01.181768 4603 scope.go:122] "RemoveContainer" containerID="48c8f9a8d6282d8d3da637130d40df86968d2eacddf7738656db68f36da975a2" Oct 30 18:33:01.182022 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:33:01.182001 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48c8f9a8d6282d8d3da637130d40df86968d2eacddf7738656db68f36da975a2\": container with ID starting with 48c8f9a8d6282d8d3da637130d40df86968d2eacddf7738656db68f36da975a2 not found: ID does not exist" containerID="48c8f9a8d6282d8d3da637130d40df86968d2eacddf7738656db68f36da975a2" Oct 30 18:33:01.182098 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:01.182033 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48c8f9a8d6282d8d3da637130d40df86968d2eacddf7738656db68f36da975a2"} err="failed to get container status \"48c8f9a8d6282d8d3da637130d40df86968d2eacddf7738656db68f36da975a2\": rpc error: code = NotFound desc = could not find container \"48c8f9a8d6282d8d3da637130d40df86968d2eacddf7738656db68f36da975a2\": container with ID starting with 48c8f9a8d6282d8d3da637130d40df86968d2eacddf7738656db68f36da975a2 not found: ID does not exist" Oct 30 18:33:01.211531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:01.211495 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6"] Oct 30 18:33:01.222846 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:01.222825 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77db82-69799db966-g2qw6"] Oct 30 18:33:01.248173 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:01.248155 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw"] Oct 30 18:33:01.251799 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:01.251780 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-5f65f58c47-fxmfw"] Oct 30 18:33:01.411896 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:01.411836 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f5e20ee-f725-4245-be00-4df9759c4a50" path="/var/lib/kubelet/pods/2f5e20ee-f725-4245-be00-4df9759c4a50/volumes" Oct 30 18:33:01.412274 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:01.412261 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="803ab509-6175-4535-a6b5-29bf877e9d72" path="/var/lib/kubelet/pods/803ab509-6175-4535-a6b5-29bf877e9d72/volumes" Oct 30 18:33:01.797546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:01.788763 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:03.495536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:03.485988 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:05.168012 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:05.167980 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:06.798252 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:06.798206 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:08.370467 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:08.370430 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:09.435090 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.435061 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb"] Oct 30 18:33:09.435753 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.435738 4603 cpu_manager.go:414] "containerMap: removing container" podUID="2f5e20ee-f725-4245-be00-4df9759c4a50" containerName="payload-processing" Oct 30 18:33:09.435813 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.435754 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f5e20ee-f725-4245-be00-4df9759c4a50" containerName="payload-processing" Oct 30 18:33:09.435813 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.435795 4603 cpu_manager.go:414] "containerMap: removing container" podUID="803ab509-6175-4535-a6b5-29bf877e9d72" containerName="payload-pre-processing" Oct 30 18:33:09.435813 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.435802 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="803ab509-6175-4535-a6b5-29bf877e9d72" containerName="payload-pre-processing" Oct 30 18:33:09.435978 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.435966 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f5e20ee-f725-4245-be00-4df9759c4a50" containerName="payload-processing" Oct 30 18:33:09.436020 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.435981 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="803ab509-6175-4535-a6b5-29bf877e9d72" containerName="payload-pre-processing" Oct 30 18:33:09.441688 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.441667 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:09.463137 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.463103 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb"] Oct 30 18:33:09.471454 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.471430 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c8eef3\"/\"auth-test-model-c8eef3-kserve-self-signed-certs\"" Oct 30 18:33:09.471771 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.471746 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c8eef3\"/\"kube-root-ca.crt\"" Oct 30 18:33:09.471892 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.471834 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c8eef3\"/\"openshift-service-ca.crt\"" Oct 30 18:33:09.471966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.471933 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c8eef3\"/\"default-dockercfg-vz7tc\"" Oct 30 18:33:09.562074 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.562047 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-model-cache\") pod \"auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:09.562232 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.562100 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-home\") pod \"auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:09.562278 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.562246 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-dshm\") pod \"auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:09.562405 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.562385 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-tmp-dir\") pod \"auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:09.562453 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.562440 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-tls-certs\") pod \"auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:09.562523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.562481 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgswl\" (UniqueName: \"kubernetes.io/projected/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-kube-api-access-dgswl\") pod \"auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:09.567102 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.567077 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:09.663685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.663660 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-dshm\") pod \"auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:09.663812 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.663756 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-tmp-dir\") pod \"auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:09.663812 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.663804 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-tls-certs\") pod \"auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:09.663904 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.663843 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgswl\" (UniqueName: \"kubernetes.io/projected/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-kube-api-access-dgswl\") pod \"auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:09.663948 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.663905 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-model-cache\") pod \"auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:09.663991 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.663953 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-home\") pod \"auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:09.664165 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.664141 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-tmp-dir\") pod \"auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:09.664276 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.664249 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-model-cache\") pod \"auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:09.664358 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.664336 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-home\") pod \"auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:09.666320 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.666300 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-dshm\") pod \"auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:09.666421 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.666401 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-tls-certs\") pod \"auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:09.703966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.703893 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgswl\" (UniqueName: \"kubernetes.io/projected/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-kube-api-access-dgswl\") pod \"auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:09.759541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:09.759179 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:10.260448 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:10.260418 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb"] Oct 30 18:33:10.260720 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:33:10.260689 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcdfbbbe_62bd_44c0_806b_d8bdbc50729b.slice/crio-4fbf06616d8af6d381f0946c4b3da270578be790f518e2f6de6cad4ad5f75734 WatchSource:0}: Error finding container 4fbf06616d8af6d381f0946c4b3da270578be790f518e2f6de6cad4ad5f75734: Status 404 returned error can't find the container with id 4fbf06616d8af6d381f0946c4b3da270578be790f518e2f6de6cad4ad5f75734 Oct 30 18:33:10.261082 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:10.261056 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:10.378530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:10.356735 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" event={"ID":"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b","Type":"ContainerStarted","Data":"4fbf06616d8af6d381f0946c4b3da270578be790f518e2f6de6cad4ad5f75734"} Oct 30 18:33:11.367079 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:11.367044 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" event={"ID":"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b","Type":"ContainerStarted","Data":"b9aa92a5eb0ef6c1c056b9298aaab7c07391e5fc8c72452a956c6ad34dfe8d98"} Oct 30 18:33:11.408632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:11.408581 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podStartSLOduration=2.408564706 podStartE2EDuration="2.408564706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:11.407954184 +0000 UTC m=+2452.123524058" watchObservedRunningTime="2026-10-30 18:33:11.408564706 +0000 UTC m=+2452.124134583" Oct 30 18:33:11.760812 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:11.760731 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:12.055558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:12.032092 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:13.693171 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:13.693132 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:15.489012 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:15.488970 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:17.188454 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:17.188392 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:17.783697 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:17.783661 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:18.453117 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:18.453083 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:18.662746 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:18.662707 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:33:19.066914 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:19.063646 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:19.760391 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:19.760361 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:19.774935 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:19.774912 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:20.555785 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:20.555754 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:21.067526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:21.058422 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:22.339971 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:33:22.339921 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:33:22.415692 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:22.415644 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:22.416301 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:22.416275 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:33:22.495718 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:22.495668 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:22.617719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:22.617615 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:22.790528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:22.769218 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:22.857207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:22.854653 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:33:22.958733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:22.958695 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:33:22.959045 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:22.959016 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" podUID="d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.037528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:23.037411 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.104117 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:23.103615 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:33:23.104117 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:23.103683 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" podUID="f7882513-ef6d-4b11-871e-39a29ad022df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.180717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:23.180075 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.275889 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:23.273620 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.329223 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:23.328685 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:33:23.329223 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:23.328773 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" podUID="58e2469b-dc0d-40ab-bd74-a47c4a9e438b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.404816 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:23.404752 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.414540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:23.413848 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.421539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:23.417436 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.422667 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:23.422261 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.543662 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:23.525781 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.877606 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:23.839729 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:33:23.877606 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:23.841341 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:33:23.877606 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:23.861568 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:33:23.882526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:23.880172 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:33:23.882526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:23.880236 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:33:23.882526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:23.880276 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:23.882526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:23.880310 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:33:23.882526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:23.880343 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:33:23.882526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:23.880377 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:33:24.117525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:24.115423 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:33:24.141585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:24.135044 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:33:24.211543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:24.211470 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:24.237719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:24.237688 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:33:24.587540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:24.573578 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": context deadline exceeded" Oct 30 18:33:24.587540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:24.575306 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:33:24.587540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:24.577180 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:33:24.625531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:24.589133 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:33:24.824530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:24.799623 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": context deadline exceeded" Oct 30 18:33:25.005577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:24.973383 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:25.151599 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:25.091050 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:25.373620 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:25.128996 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:25.374093 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:25.256875 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:33:25.374423 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:25.308245 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:33:25.374585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:25.319669 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:33:25.486750 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:25.323737 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:25.620641 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:25.610920 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Oct 30 18:33:25.621314 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:25.612612 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:25.633575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:25.613287 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:33:25.633575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:25.613574 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:33:25.769855 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:25.723626 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Oct 30 18:33:25.769855 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:25.758912 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:33:25.850649 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:25.850594 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": context deadline exceeded" Oct 30 18:33:25.889491 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:25.871115 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": context deadline exceeded" Oct 30 18:33:26.160577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:26.158595 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:26.375695 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:26.375640 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:26.397575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:26.397490 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:33:26.633528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:26.631161 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:33:26.641528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:26.639731 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:33:26.687190 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:26.682739 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": context deadline exceeded" Oct 30 18:33:26.795531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:26.790675 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:33:26.812804 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:26.812750 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:33:26.935590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:26.894966 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": context deadline exceeded" Oct 30 18:33:27.001611 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:26.925410 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:33:27.001611 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:26.925805 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/health\": context deadline exceeded" Oct 30 18:33:27.001611 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:26.925727 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:27.026378 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:27.026347 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:27.139442 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:27.135650 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:27.160911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:27.160869 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:27.354529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:27.351697 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded" Oct 30 18:33:27.380529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:27.379588 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:27.464529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:27.457139 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:33:27.634581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:27.631485 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:27.673587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:27.663623 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:27.695528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:27.690807 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:27.802531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:27.798462 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:27.845650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:27.845582 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:27.983805 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:27.983679 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:27.992669 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:27.992587 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:33:27.993960 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:27.993893 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": context deadline exceeded" Oct 30 18:33:28.096542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:28.080590 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:28.180321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:28.177796 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:28.480530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:28.478467 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:28.507528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:28.507354 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:28.511537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:28.507803 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:33:28.614771 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:28.614696 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Oct 30 18:33:28.642758 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:28.642708 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:28.671536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:28.667662 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:28.702346 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:28.702302 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:28.813422 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:28.807598 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:28.884166 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:28.856048 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.014533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:28.989878 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": context deadline exceeded" Oct 30 18:33:29.015130 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:29.013216 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": context deadline exceeded" Oct 30 18:33:29.015954 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:29.013494 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.091701 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:29.091610 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:33:29.091984 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:29.091954 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" podUID="22e0b280-2b65-4fad-8344-67049190d108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.184799 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:29.184754 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.517214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:29.517108 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.518127 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:29.518094 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.688536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:29.673468 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.717635 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:29.675493 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.717635 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:29.714627 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.827530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:29.821091 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:33:29.856991 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:29.856951 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:33:29.888533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:29.863587 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:33:29.888533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:29.863646 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" podUID="81f49829-73ed-4760-9444-f1793861eba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.027569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:30.026411 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": context deadline exceeded" Oct 30 18:33:30.027569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:30.026857 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": context deadline exceeded" Oct 30 18:33:30.027569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:30.027176 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:33:30.082602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:30.079612 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:33:30.082602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:30.079680 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" podUID="571fc9dd-71e3-4fa7-b1af-e74f0a96654d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.192529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:30.191601 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.297531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:30.293590 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.518861 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:30.518719 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.652525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:30.640183 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.684431 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:30.684389 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.707533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:30.702634 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.733528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:30.732604 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.813578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:30.800674 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.828530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:30.824970 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.862532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:30.860935 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:31.029785 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:31.029738 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:31.030484 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:31.030450 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": context deadline exceeded" Oct 30 18:33:31.031008 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:31.030982 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": context deadline exceeded" Oct 30 18:33:31.216596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:31.215605 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:33:31.258566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:31.244198 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:31.537534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:31.521751 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:33:31.649914 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:31.649858 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:31.687658 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:31.687593 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:33:31.711837 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:31.711780 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:31.742975 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:31.742931 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": context deadline exceeded" Oct 30 18:33:31.829742 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:31.829631 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:31.862529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:31.861126 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:32.031655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:32.031604 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:32.032540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:32.032073 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": context deadline exceeded" Oct 30 18:33:32.033803 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:32.032104 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:33:32.219097 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:32.218840 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:33:32.524894 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:32.524853 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:33:32.685529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:32.658652 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:32.693595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:32.693550 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:32.735619 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:32.720656 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:33:32.743630 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:32.743579 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:32.842587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:32.834833 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:33:32.863539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:32.862589 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:33.041149 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:33.040387 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": context deadline exceeded" Oct 30 18:33:33.041149 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:33.040630 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": context deadline exceeded" Oct 30 18:33:33.060528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:33.057666 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:33.233113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:33.232988 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:33:33.574148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:33.574098 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:33:33.574529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:33.574483 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" podUID="ec3c95aa-215c-4f15-8dc8-bc0beb954236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:33.586230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:33.574415 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:33.697716 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:33.697675 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:33.699115 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:33.698087 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:33.721919 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:33.721869 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:33:33.744539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:33.733849 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:33.745271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:33.745236 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": context deadline exceeded" Oct 30 18:33:33.848856 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:33.848760 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:33:33.870319 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:33.870280 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:33:34.052206 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:34.052166 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": context deadline exceeded" Oct 30 18:33:34.072645 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:34.072601 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:33:34.075895 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:34.075715 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:33:34.078489 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:34.078458 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:34.096531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:34.093275 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:33:34.096531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:34.093327 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" podUID="187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:34.242300 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:34.242209 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:33:34.307830 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:34.307794 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:33:34.308139 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:34.308096 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" podUID="b6e43478-8cc1-4452-8bc8-fed14f127896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:34.587655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:34.587613 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:34.626157 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:34.626115 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:34.699617 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:34.699574 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:33:34.712282 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:34.703816 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:33:34.723537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:34.722432 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:34.750527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:34.747590 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": context deadline exceeded" Oct 30 18:33:34.761901 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:34.760329 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:33:34.877569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:34.876966 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:33:34.886008 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:34.885218 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:34.953182 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:34.945531 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:33:34.984562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:34.976906 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:34.984562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:34.984080 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:33:35.003529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:35.002053 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:33:35.007525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:35.005218 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:33:35.008496 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:35.008471 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:33:35.018541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:35.014696 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:33:35.047648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:35.047614 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:33:35.065565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:35.065537 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:33:36.040554 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:36.040519 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:37.072761 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:37.072725 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:37.526097 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:37.526068 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:37.749231 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:37.749195 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:39.459226 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:39.459181 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:41.149061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:41.149018 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:41.868790 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:41.868749 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd"] Oct 30 18:33:41.876100 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:41.876073 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:41.914749 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:41.914713 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd"] Oct 30 18:33:41.915467 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:41.915435 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:41.915632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:41.915563 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:41.915701 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:41.915629 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:41.915701 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:41.915667 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:41.915821 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:41.915802 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2888d82-f0fe-49af-a23d-c97b4f30be70-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:41.915909 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:41.915847 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9r5p\" (UniqueName: \"kubernetes.io/projected/d2888d82-f0fe-49af-a23d-c97b4f30be70-kube-api-access-n9r5p\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:41.915963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:41.915918 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:41.916615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:41.916591 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:41.916919 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:41.916898 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:42.018632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.018602 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:42.018798 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.018644 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:42.018798 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.018696 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:42.018798 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.018743 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:42.018946 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.018829 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:42.018986 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.018937 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:42.018986 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.018979 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:42.019182 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.019161 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2888d82-f0fe-49af-a23d-c97b4f30be70-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:42.019278 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.019181 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:42.019278 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.019216 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9r5p\" (UniqueName: \"kubernetes.io/projected/d2888d82-f0fe-49af-a23d-c97b4f30be70-kube-api-access-n9r5p\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:42.019398 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.019319 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:42.019458 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.019408 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:42.019458 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.019435 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:42.019973 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.019944 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2888d82-f0fe-49af-a23d-c97b4f30be70-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:42.021665 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.021636 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:42.022125 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.022108 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:42.060168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.060142 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9r5p\" (UniqueName: \"kubernetes.io/projected/d2888d82-f0fe-49af-a23d-c97b4f30be70-kube-api-access-n9r5p\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:42.087755 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.087725 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:42.192643 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.192562 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:42.530175 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.530151 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd"] Oct 30 18:33:42.531873 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:33:42.531843 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2888d82_f0fe_49af_a23d_c97b4f30be70.slice/crio-2f083e0c5f2d5823c20eb23d2999fbb8819e0821632e480a83498dad2c93c0fa WatchSource:0}: Error finding container 2f083e0c5f2d5823c20eb23d2999fbb8819e0821632e480a83498dad2c93c0fa: Status 404 returned error can't find the container with id 2f083e0c5f2d5823c20eb23d2999fbb8819e0821632e480a83498dad2c93c0fa Oct 30 18:33:42.534814 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.534780 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:33:42.534921 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.534860 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:33:42.534921 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.534908 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:33:42.998686 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.998585 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" event={"ID":"d2888d82-f0fe-49af-a23d-c97b4f30be70","Type":"ContainerStarted","Data":"88c2c26dd336347a45f793eda01f76484b89023ff05551b64239f67b904da295"} Oct 30 18:33:42.998686 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:42.998650 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" event={"ID":"d2888d82-f0fe-49af-a23d-c97b4f30be70","Type":"ContainerStarted","Data":"2f083e0c5f2d5823c20eb23d2999fbb8819e0821632e480a83498dad2c93c0fa"} Oct 30 18:33:43.000755 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:43.000731 4603 generic.go:356] "Generic (PLEG): container finished" podID="0815ac39-de71-4bc3-98d3-28c88a9d2139" containerID="7e367247a2f0e8c781fafc99df6353df88fab7c94e13841a9887e8c5c86603da" exitCode=0 Oct 30 18:33:43.000909 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:43.000803 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" event={"ID":"0815ac39-de71-4bc3-98d3-28c88a9d2139","Type":"ContainerDied","Data":"7e367247a2f0e8c781fafc99df6353df88fab7c94e13841a9887e8c5c86603da"} Oct 30 18:33:43.001289 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:43.001274 4603 scope.go:122] "RemoveContainer" containerID="7e367247a2f0e8c781fafc99df6353df88fab7c94e13841a9887e8c5c86603da" Oct 30 18:33:43.003040 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:43.003008 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:33:43.003156 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:43.003075 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:33:43.003156 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:43.003109 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:33:43.059976 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:43.059835 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" podStartSLOduration=2.059818093 podStartE2EDuration="2.059818093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:43.058172301 +0000 UTC m=+2483.773742176" watchObservedRunningTime="2026-10-30 18:33:43.059818093 +0000 UTC m=+2483.775387967" Oct 30 18:33:43.108029 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:43.107981 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:43.312647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:43.276992 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:43.312647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:43.277262 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 18:33:43.312647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:43.277310 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" podUID="d2888d82-f0fe-49af-a23d-c97b4f30be70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 18:33:43.479885 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:43.479852 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:33:44.048937 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:44.048888 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" event={"ID":"0815ac39-de71-4bc3-98d3-28c88a9d2139","Type":"ContainerStarted","Data":"5392e5dfd5c8f6e03d4c032efc9f20da836316906e9e17868e71835c31ef1515"} Oct 30 18:33:44.226529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:44.209945 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 18:33:44.226529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:44.210009 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" podUID="d2888d82-f0fe-49af-a23d-c97b4f30be70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 18:33:44.242604 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:44.234301 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:33:44.243738 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:44.243691 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 18:33:44.244028 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:44.243999 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" podUID="0815ac39-de71-4bc3-98d3-28c88a9d2139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 18:33:44.281765 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:44.281734 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:44.467140 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:44.466911 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:44.641308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:44.641225 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx"] Oct 30 18:33:44.641866 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:44.641835 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" podUID="a30517af-31fe-458a-8b17-40bc5ab61274" containerName="istio-proxy" containerID="cri-o://3dd642d13619bdb151fe0bb4378962353668ac597acbf8f8af8bfe22eb012bd7" gracePeriod=30 Oct 30 18:33:44.664521 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:44.661309 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj"] Oct 30 18:33:44.664521 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:44.661561 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" podUID="a065f551-ec87-4081-bc18-27d0dd4234e3" containerName="istio-proxy" containerID="cri-o://50eaa51974dd5b8bb5f841a01c69f75f6e19d94f457d45c2ce928e68b17fb26e" gracePeriod=30 Oct 30 18:33:44.666092 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:44.665995 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:44.668175 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:44.667652 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8"] Oct 30 18:33:44.668175 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:44.667860 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerName="istio-proxy" containerID="cri-o://db1c2d3bdf6b84e095aebc306c15e9f4cbdf6b12f94e8a22df62f7a316156ab0" gracePeriod=30 Oct 30 18:33:45.253789 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.193142 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 18:33:45.253789 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.193201 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" podUID="d2888d82-f0fe-49af-a23d-c97b4f30be70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 18:33:45.253789 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.216720 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 18:33:45.253789 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.216785 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" podUID="0815ac39-de71-4bc3-98d3-28c88a9d2139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 18:33:45.509591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.507343 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:45.706817 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.706786 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p"] Oct 30 18:33:45.760590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.759618 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p"] Oct 30 18:33:45.760590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.759762 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.777448 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.777419 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-5cb3d5-openshift-default-dockercfg-69d2c\"" Oct 30 18:33:45.842527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.834079 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-workload-socket\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.842527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.834187 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-credential-socket\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.842527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.834327 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-envoy\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.842527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.834427 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtmtm\" (UniqueName: \"kubernetes.io/projected/a5a9b344-c8bb-433b-a6a2-e62c357f4101-kube-api-access-gtmtm\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.842527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.834598 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-token\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.842527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.834679 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-data\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.842527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.834706 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-podinfo\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.842527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.834795 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-workload-certs\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.842527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.834853 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istiod-ca-cert\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.947548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.945028 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-data\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.947548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.944475 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-data\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.947548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.945151 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-podinfo\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.947548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.946184 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-workload-certs\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.947548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.946306 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-workload-certs\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.947548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.946399 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istiod-ca-cert\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.947548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.947349 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istiod-ca-cert\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.947548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.947537 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-workload-socket\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.948096 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.947579 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-credential-socket\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.948096 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.947633 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-envoy\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.948096 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.947700 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtmtm\" (UniqueName: \"kubernetes.io/projected/a5a9b344-c8bb-433b-a6a2-e62c357f4101-kube-api-access-gtmtm\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.948096 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.947878 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-token\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.948096 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.947887 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-credential-socket\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.948380 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.948293 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-workload-socket\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.949525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.949480 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-podinfo\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.950834 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.950790 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-envoy\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.981298 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.980233 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtmtm\" (UniqueName: \"kubernetes.io/projected/a5a9b344-c8bb-433b-a6a2-e62c357f4101-kube-api-access-gtmtm\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:45.981298 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:45.980254 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-token\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:46.086147 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:46.086119 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:46.199531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:46.197677 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 18:33:46.199531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:46.197754 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" podUID="d2888d82-f0fe-49af-a23d-c97b4f30be70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 18:33:46.225634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:46.218913 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 18:33:46.225634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:46.218966 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" podUID="0815ac39-de71-4bc3-98d3-28c88a9d2139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 18:33:46.544015 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:33:46.543973 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a9b344_c8bb_433b_a6a2_e62c357f4101.slice/crio-8ae379dbf029551bf3bc963e83f05143db05fd8d8d56ab23e74c7f5c8bcc1ca2 WatchSource:0}: Error finding container 8ae379dbf029551bf3bc963e83f05143db05fd8d8d56ab23e74c7f5c8bcc1ca2: Status 404 returned error can't find the container with id 8ae379dbf029551bf3bc963e83f05143db05fd8d8d56ab23e74c7f5c8bcc1ca2 Oct 30 18:33:46.544781 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:46.544543 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p"] Oct 30 18:33:46.547637 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:46.547601 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:33:46.547726 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:46.547710 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:33:46.547777 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:46.547752 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:33:47.193542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:47.188742 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" event={"ID":"a5a9b344-c8bb-433b-a6a2-e62c357f4101","Type":"ContainerStarted","Data":"3f3e259ff991772ad99c365783dc5153ea191f933d17f18d3da276bbd70171de"} Oct 30 18:33:47.193542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:47.188812 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" event={"ID":"a5a9b344-c8bb-433b-a6a2-e62c357f4101","Type":"ContainerStarted","Data":"8ae379dbf029551bf3bc963e83f05143db05fd8d8d56ab23e74c7f5c8bcc1ca2"} Oct 30 18:33:47.199913 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:47.196863 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 18:33:47.199913 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:47.196916 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" podUID="d2888d82-f0fe-49af-a23d-c97b4f30be70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 18:33:47.226527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:47.225338 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 18:33:47.226527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:47.225391 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" podUID="0815ac39-de71-4bc3-98d3-28c88a9d2139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 18:33:47.249709 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:47.249645 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" podStartSLOduration=2.249624454 podStartE2EDuration="2.249624454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:47.238118085 +0000 UTC m=+2487.953687962" watchObservedRunningTime="2026-10-30 18:33:47.249624454 +0000 UTC m=+2487.965194328" Oct 30 18:33:47.671530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:47.660255 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:48.087002 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:48.086964 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:48.095538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:48.088598 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 18:33:48.095538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:48.088657 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" podUID="a5a9b344-c8bb-433b-a6a2-e62c357f4101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 18:33:48.207532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:48.204950 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 18:33:48.207532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:48.205014 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" podUID="d2888d82-f0fe-49af-a23d-c97b4f30be70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 18:33:48.232531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:48.230003 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 18:33:48.232531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:48.230079 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" podUID="0815ac39-de71-4bc3-98d3-28c88a9d2139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 18:33:49.086975 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:49.086942 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 18:33:49.087396 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:49.086991 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" podUID="a5a9b344-c8bb-433b-a6a2-e62c357f4101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 18:33:49.206915 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:49.206875 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 18:33:49.207197 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:49.207169 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" podUID="d2888d82-f0fe-49af-a23d-c97b4f30be70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 18:33:49.230906 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:49.230870 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 18:33:49.231184 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:49.231155 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" podUID="0815ac39-de71-4bc3-98d3-28c88a9d2139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 18:33:49.252972 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:49.252810 4603 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:33:49.253247 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:49.253222 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" podUID="a065f551-ec87-4081-bc18-27d0dd4234e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:33:49.677085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:49.677018 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:50.087636 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.087035 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 18:33:50.087636 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.087096 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" podUID="a5a9b344-c8bb-433b-a6a2-e62c357f4101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 18:33:50.225593 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.222339 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 18:33:50.225593 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.222403 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" podUID="0815ac39-de71-4bc3-98d3-28c88a9d2139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 18:33:50.277146 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.277109 4603 generic.go:356] "Generic (PLEG): container finished" podID="a065f551-ec87-4081-bc18-27d0dd4234e3" containerID="50eaa51974dd5b8bb5f841a01c69f75f6e19d94f457d45c2ce928e68b17fb26e" exitCode=0 Oct 30 18:33:50.277444 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.277419 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" event={"ID":"a065f551-ec87-4081-bc18-27d0dd4234e3","Type":"ContainerDied","Data":"50eaa51974dd5b8bb5f841a01c69f75f6e19d94f457d45c2ce928e68b17fb26e"} Oct 30 18:33:50.280152 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.280122 4603 generic.go:356] "Generic (PLEG): container finished" podID="a30517af-31fe-458a-8b17-40bc5ab61274" containerID="3dd642d13619bdb151fe0bb4378962353668ac597acbf8f8af8bfe22eb012bd7" exitCode=0 Oct 30 18:33:50.280546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.280520 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" event={"ID":"a30517af-31fe-458a-8b17-40bc5ab61274","Type":"ContainerDied","Data":"3dd642d13619bdb151fe0bb4378962353668ac597acbf8f8af8bfe22eb012bd7"} Oct 30 18:33:50.283561 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.283538 4603 generic.go:356] "Generic (PLEG): container finished" podID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerID="db1c2d3bdf6b84e095aebc306c15e9f4cbdf6b12f94e8a22df62f7a316156ab0" exitCode=0 Oct 30 18:33:50.283786 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.283766 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" event={"ID":"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af","Type":"ContainerDied","Data":"db1c2d3bdf6b84e095aebc306c15e9f4cbdf6b12f94e8a22df62f7a316156ab0"} Oct 30 18:33:50.283943 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.283928 4603 scope.go:122] "RemoveContainer" containerID="c78b43d96281f3f140142fdc01deb094700395b4c1b15ddc1bc01e2b05f70932" Oct 30 18:33:50.340617 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.334415 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:50.340617 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.336109 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:50.340617 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.337854 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:33:50.474214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.474177 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:33:50.566408 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.566381 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:33:50.571115 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.569747 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:33:50.645574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.644795 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-istio-envoy\") pod \"a30517af-31fe-458a-8b17-40bc5ab61274\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " Oct 30 18:33:50.645574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.644850 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a30517af-31fe-458a-8b17-40bc5ab61274-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30517af-31fe-458a-8b17-40bc5ab61274-istiod-ca-cert\") pod \"a30517af-31fe-458a-8b17-40bc5ab61274\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " Oct 30 18:33:50.645574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.645067 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-credential-socket\") pod \"a30517af-31fe-458a-8b17-40bc5ab61274\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " Oct 30 18:33:50.645574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.645126 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a30517af-31fe-458a-8b17-40bc5ab61274-kube-api-access-w8jdz\" (UniqueName: \"kubernetes.io/projected/a30517af-31fe-458a-8b17-40bc5ab61274-kube-api-access-w8jdz\") pod \"a30517af-31fe-458a-8b17-40bc5ab61274\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " Oct 30 18:33:50.645574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.645159 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-workload-certs\") pod \"a30517af-31fe-458a-8b17-40bc5ab61274\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " Oct 30 18:33:50.645574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.645218 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-istio-data\") pod \"a30517af-31fe-458a-8b17-40bc5ab61274\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " Oct 30 18:33:50.645574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.645267 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a30517af-31fe-458a-8b17-40bc5ab61274-istio-token\" (UniqueName: \"kubernetes.io/projected/a30517af-31fe-458a-8b17-40bc5ab61274-istio-token\") pod \"a30517af-31fe-458a-8b17-40bc5ab61274\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " Oct 30 18:33:50.645574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.645325 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-workload-socket\") pod \"a30517af-31fe-458a-8b17-40bc5ab61274\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " Oct 30 18:33:50.645574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.645494 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a30517af-31fe-458a-8b17-40bc5ab61274-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30517af-31fe-458a-8b17-40bc5ab61274-istio-podinfo\") pod \"a30517af-31fe-458a-8b17-40bc5ab61274\" (UID: \"a30517af-31fe-458a-8b17-40bc5ab61274\") " Oct 30 18:33:50.649239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.648941 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-workload-certs" pod "a30517af-31fe-458a-8b17-40bc5ab61274" (UID: "a30517af-31fe-458a-8b17-40bc5ab61274"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.650653 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.650621 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-istio-envoy" pod "a30517af-31fe-458a-8b17-40bc5ab61274" (UID: "a30517af-31fe-458a-8b17-40bc5ab61274"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.650880 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.650854 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-workload-socket" pod "a30517af-31fe-458a-8b17-40bc5ab61274" (UID: "a30517af-31fe-458a-8b17-40bc5ab61274"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.652075 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.652037 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a30517af-31fe-458a-8b17-40bc5ab61274-istio-podinfo" pod "a30517af-31fe-458a-8b17-40bc5ab61274" (UID: "a30517af-31fe-458a-8b17-40bc5ab61274"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:33:50.655779 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.655751 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a30517af-31fe-458a-8b17-40bc5ab61274-istiod-ca-cert" pod "a30517af-31fe-458a-8b17-40bc5ab61274" (UID: "a30517af-31fe-458a-8b17-40bc5ab61274"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:50.659746 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.659646 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-credential-socket" pod "a30517af-31fe-458a-8b17-40bc5ab61274" (UID: "a30517af-31fe-458a-8b17-40bc5ab61274"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.659746 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.659712 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-istio-data" pod "a30517af-31fe-458a-8b17-40bc5ab61274" (UID: "a30517af-31fe-458a-8b17-40bc5ab61274"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.663618 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.661434 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a30517af-31fe-458a-8b17-40bc5ab61274-istio-token" pod "a30517af-31fe-458a-8b17-40bc5ab61274" (UID: "a30517af-31fe-458a-8b17-40bc5ab61274"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:50.663618 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.661588 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a30517af-31fe-458a-8b17-40bc5ab61274-kube-api-access-w8jdz" pod "a30517af-31fe-458a-8b17-40bc5ab61274" (UID: "a30517af-31fe-458a-8b17-40bc5ab61274"). InnerVolumeSpecName "kube-api-access-w8jdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:50.748271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.748126 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-token\" (UniqueName: \"kubernetes.io/projected/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-token\") pod \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " Oct 30 18:33:50.748271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.748189 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-workload-certs\") pod \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " Oct 30 18:33:50.748475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.748337 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-workload-socket\") pod \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " Oct 30 18:33:50.748475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.748403 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-envoy\") pod \"a065f551-ec87-4081-bc18-27d0dd4234e3\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " Oct 30 18:33:50.748475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.748464 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-envoy\") pod \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " Oct 30 18:33:50.748746 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.748577 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-kube-api-access-fjqcb\" (UniqueName: \"kubernetes.io/projected/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-kube-api-access-fjqcb\") pod \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " Oct 30 18:33:50.748746 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.748624 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-podinfo\") pod \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " Oct 30 18:33:50.748746 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.748660 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-credential-socket\") pod \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " Oct 30 18:33:50.748985 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.748749 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-data\") pod \"a065f551-ec87-4081-bc18-27d0dd4234e3\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " Oct 30 18:33:50.748985 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.748792 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-token\" (UniqueName: \"kubernetes.io/projected/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-token\") pod \"a065f551-ec87-4081-bc18-27d0dd4234e3\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " Oct 30 18:33:50.748985 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.748840 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-data\") pod \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " Oct 30 18:33:50.748985 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.748882 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-credential-socket\") pod \"a065f551-ec87-4081-bc18-27d0dd4234e3\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " Oct 30 18:33:50.748985 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.748921 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a065f551-ec87-4081-bc18-27d0dd4234e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a065f551-ec87-4081-bc18-27d0dd4234e3-istiod-ca-cert\") pod \"a065f551-ec87-4081-bc18-27d0dd4234e3\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " Oct 30 18:33:50.749347 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.748994 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istiod-ca-cert\") pod \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\" (UID: \"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af\") " Oct 30 18:33:50.749347 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.749051 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-workload-socket\") pod \"a065f551-ec87-4081-bc18-27d0dd4234e3\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " Oct 30 18:33:50.749347 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.749122 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-podinfo\") pod \"a065f551-ec87-4081-bc18-27d0dd4234e3\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " Oct 30 18:33:50.749347 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.749178 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a065f551-ec87-4081-bc18-27d0dd4234e3-kube-api-access-tvjjs\" (UniqueName: \"kubernetes.io/projected/a065f551-ec87-4081-bc18-27d0dd4234e3-kube-api-access-tvjjs\") pod \"a065f551-ec87-4081-bc18-27d0dd4234e3\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " Oct 30 18:33:50.749347 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.749262 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-workload-certs\") pod \"a065f551-ec87-4081-bc18-27d0dd4234e3\" (UID: \"a065f551-ec87-4081-bc18-27d0dd4234e3\") " Oct 30 18:33:50.750668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.750639 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.750801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.750673 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8jdz\" (UniqueName: \"kubernetes.io/projected/a30517af-31fe-458a-8b17-40bc5ab61274-kube-api-access-w8jdz\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.750801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.750689 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.750801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.750709 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.750801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.750723 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30517af-31fe-458a-8b17-40bc5ab61274-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.750801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.750737 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.750801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.750751 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30517af-31fe-458a-8b17-40bc5ab61274-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.751142 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.750769 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30517af-31fe-458a-8b17-40bc5ab61274-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.751142 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.750821 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30517af-31fe-458a-8b17-40bc5ab61274-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.754313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.754282 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-data" pod "85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" (UID: "85f4e2fe-260f-49bd-9c9f-dc73eff6c5af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.754549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.754480 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-workload-certs" pod "85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" (UID: "85f4e2fe-260f-49bd-9c9f-dc73eff6c5af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.754734 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.754712 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-workload-socket" pod "85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" (UID: "85f4e2fe-260f-49bd-9c9f-dc73eff6c5af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.755440 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.755418 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-token" pod "85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" (UID: "85f4e2fe-260f-49bd-9c9f-dc73eff6c5af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:50.755688 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.755666 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-credential-socket" pod "a065f551-ec87-4081-bc18-27d0dd4234e3" (UID: "a065f551-ec87-4081-bc18-27d0dd4234e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.755980 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.755958 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a065f551-ec87-4081-bc18-27d0dd4234e3-istiod-ca-cert" pod "a065f551-ec87-4081-bc18-27d0dd4234e3" (UID: "a065f551-ec87-4081-bc18-27d0dd4234e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:50.756326 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.756303 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istiod-ca-cert" pod "85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" (UID: "85f4e2fe-260f-49bd-9c9f-dc73eff6c5af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:50.756569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.756550 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-workload-socket" pod "a065f551-ec87-4081-bc18-27d0dd4234e3" (UID: "a065f551-ec87-4081-bc18-27d0dd4234e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.759176 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.759146 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-podinfo" pod "a065f551-ec87-4081-bc18-27d0dd4234e3" (UID: "a065f551-ec87-4081-bc18-27d0dd4234e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:33:50.761807 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.761778 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-workload-certs" pod "a065f551-ec87-4081-bc18-27d0dd4234e3" (UID: "a065f551-ec87-4081-bc18-27d0dd4234e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.762050 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.762016 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-envoy" pod "a065f551-ec87-4081-bc18-27d0dd4234e3" (UID: "a065f551-ec87-4081-bc18-27d0dd4234e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.764208 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.764003 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-data" pod "a065f551-ec87-4081-bc18-27d0dd4234e3" (UID: "a065f551-ec87-4081-bc18-27d0dd4234e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.764208 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.764025 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-podinfo" pod "85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" (UID: "85f4e2fe-260f-49bd-9c9f-dc73eff6c5af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:33:50.764777 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.764733 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a065f551-ec87-4081-bc18-27d0dd4234e3-kube-api-access-tvjjs" pod "a065f551-ec87-4081-bc18-27d0dd4234e3" (UID: "a065f551-ec87-4081-bc18-27d0dd4234e3"). InnerVolumeSpecName "kube-api-access-tvjjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:50.765087 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.765061 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-credential-socket" pod "85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" (UID: "85f4e2fe-260f-49bd-9c9f-dc73eff6c5af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.766923 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.766895 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-token" pod "a065f551-ec87-4081-bc18-27d0dd4234e3" (UID: "a065f551-ec87-4081-bc18-27d0dd4234e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:50.767629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.767601 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-envoy" pod "85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" (UID: "85f4e2fe-260f-49bd-9c9f-dc73eff6c5af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.776074 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.768913 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-kube-api-access-fjqcb" pod "85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" (UID: "85f4e2fe-260f-49bd-9c9f-dc73eff6c5af"). InnerVolumeSpecName "kube-api-access-fjqcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:50.852888 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.852799 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.852888 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.852835 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.852888 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.852852 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.852888 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.852867 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvjjs\" (UniqueName: \"kubernetes.io/projected/a065f551-ec87-4081-bc18-27d0dd4234e3-kube-api-access-tvjjs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.852888 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.852882 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.853312 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.852895 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.853312 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.852910 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.853312 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.852922 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.853312 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.852935 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.853312 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.852948 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.853312 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.852961 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjqcb\" (UniqueName: \"kubernetes.io/projected/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-kube-api-access-fjqcb\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.853312 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.852974 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.853312 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.852987 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.853312 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.853000 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.853312 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.853013 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a065f551-ec87-4081-bc18-27d0dd4234e3-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.853312 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.853026 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.853312 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.853038 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a065f551-ec87-4081-bc18-27d0dd4234e3-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.853312 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:50.853052 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a065f551-ec87-4081-bc18-27d0dd4234e3-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:33:51.087467 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:51.087429 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 18:33:51.087651 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:51.087484 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" podUID="a5a9b344-c8bb-433b-a6a2-e62c357f4101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 18:33:51.221729 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:51.221654 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:33:51.222109 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:51.221968 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:33:51.222869 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:51.222848 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:33:51.318169 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:51.318136 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" Oct 30 18:33:51.318169 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:51.318149 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8" event={"ID":"85f4e2fe-260f-49bd-9c9f-dc73eff6c5af","Type":"ContainerDied","Data":"2654115544eafc6601b4ef1d5975a68fb2baa2c63fdacb10f1848a04a2cf2028"} Oct 30 18:33:51.318460 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:51.318216 4603 scope.go:122] "RemoveContainer" containerID="db1c2d3bdf6b84e095aebc306c15e9f4cbdf6b12f94e8a22df62f7a316156ab0" Oct 30 18:33:51.320336 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:51.320308 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" event={"ID":"a065f551-ec87-4081-bc18-27d0dd4234e3","Type":"ContainerDied","Data":"748a4d33053fd8e61c20b08d8f41fe2548bdb8a7ab0d6f24f3d8dc9f6ec5ac0a"} Oct 30 18:33:51.320743 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:51.320720 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj" Oct 30 18:33:51.323334 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:51.323213 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" event={"ID":"a30517af-31fe-458a-8b17-40bc5ab61274","Type":"ContainerDied","Data":"dbd3181e1e1fcdefe84c860ae2cbcc767f3792e96dd71e26000a7ec4f52f5770"} Oct 30 18:33:51.323539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:51.323523 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx" Oct 30 18:33:51.335573 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:51.334812 4603 scope.go:122] "RemoveContainer" containerID="50eaa51974dd5b8bb5f841a01c69f75f6e19d94f457d45c2ce928e68b17fb26e" Oct 30 18:33:51.352004 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:51.351987 4603 scope.go:122] "RemoveContainer" containerID="3dd642d13619bdb151fe0bb4378962353668ac597acbf8f8af8bfe22eb012bd7" Oct 30 18:33:51.376409 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:51.376383 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8"] Oct 30 18:33:51.382218 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:51.382191 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-cq9n8"] Oct 30 18:33:51.439552 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:51.439471 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" path="/var/lib/kubelet/pods/85f4e2fe-260f-49bd-9c9f-dc73eff6c5af/volumes" Oct 30 18:33:51.450846 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:51.450815 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx"] Oct 30 18:33:51.455429 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:51.455405 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-ll5wx"] Oct 30 18:33:51.487726 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:51.487666 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj"] Oct 30 18:33:51.517741 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:51.517709 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-mgqqj"] Oct 30 18:33:51.723529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:51.722476 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:51.944169 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:33:51.944131 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:33:52.087176 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:52.087148 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 18:33:52.087334 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:52.087207 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" podUID="a5a9b344-c8bb-433b-a6a2-e62c357f4101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 18:33:53.086434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:53.086398 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 18:33:53.086896 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:53.086468 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" podUID="a5a9b344-c8bb-433b-a6a2-e62c357f4101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 18:33:53.424332 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:53.424253 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a065f551-ec87-4081-bc18-27d0dd4234e3" path="/var/lib/kubelet/pods/a065f551-ec87-4081-bc18-27d0dd4234e3/volumes" Oct 30 18:33:53.424922 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:53.424904 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a30517af-31fe-458a-8b17-40bc5ab61274" path="/var/lib/kubelet/pods/a30517af-31fe-458a-8b17-40bc5ab61274/volumes" Oct 30 18:33:53.623461 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:53.623422 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:54.086485 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:54.086453 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 18:33:54.086897 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:54.086527 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" podUID="a5a9b344-c8bb-433b-a6a2-e62c357f4101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 18:33:55.092291 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:55.092262 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:55.092744 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:55.092722 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:55.093518 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:55.093482 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:33:55.430578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:55.430443 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:57.432329 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:57.432290 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:59.203535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:33:59.198461 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:00.971730 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:00.971692 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:02.397711 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:02.397672 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:02.454398 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:02.454370 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9"] Oct 30 18:34:02.454605 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:02.454586 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9" podUID="b79b5853-8745-436a-aeff-aada86fb4853" containerName="payload-processing" containerID="cri-o://4f0b6dcca8452cf73c69505ae2b28df2f25391a129e8877818f12595f4dc6bcf" gracePeriod=30 Oct 30 18:34:02.488954 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:02.488922 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk"] Oct 30 18:34:02.489212 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:02.489182 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk" podUID="591b0147-7c0a-4295-886d-9d238a646b9a" containerName="payload-pre-processing" containerID="cri-o://19a20d6097bc953ffa521639253cfb183315b904328bac6c7e403f00354d5a1d" gracePeriod=30 Oct 30 18:34:02.567667 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:02.567637 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:02.710435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:02.710363 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:03.081524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.078649 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:03.182549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.181858 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9" Oct 30 18:34:03.186291 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.186270 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk" Oct 30 18:34:03.232729 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.232659 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:03.284534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.284046 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b79b5853-8745-436a-aeff-aada86fb4853-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b79b5853-8745-436a-aeff-aada86fb4853-plugins-config-volume\") pod \"b79b5853-8745-436a-aeff-aada86fb4853\" (UID: \"b79b5853-8745-436a-aeff-aada86fb4853\") " Oct 30 18:34:03.284534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.284388 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/591b0147-7c0a-4295-886d-9d238a646b9a-kube-api-access-2jh7f\" (UniqueName: \"kubernetes.io/projected/591b0147-7c0a-4295-886d-9d238a646b9a-kube-api-access-2jh7f\") pod \"591b0147-7c0a-4295-886d-9d238a646b9a\" (UID: \"591b0147-7c0a-4295-886d-9d238a646b9a\") " Oct 30 18:34:03.284821 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.284568 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b79b5853-8745-436a-aeff-aada86fb4853-kube-api-access-lrbvj\" (UniqueName: \"kubernetes.io/projected/b79b5853-8745-436a-aeff-aada86fb4853-kube-api-access-lrbvj\") pod \"b79b5853-8745-436a-aeff-aada86fb4853\" (UID: \"b79b5853-8745-436a-aeff-aada86fb4853\") " Oct 30 18:34:03.284821 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.284678 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/591b0147-7c0a-4295-886d-9d238a646b9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591b0147-7c0a-4295-886d-9d238a646b9a-plugins-config-volume\") pod \"591b0147-7c0a-4295-886d-9d238a646b9a\" (UID: \"591b0147-7c0a-4295-886d-9d238a646b9a\") " Oct 30 18:34:03.293785 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.287327 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/591b0147-7c0a-4295-886d-9d238a646b9a-plugins-config-volume" pod "591b0147-7c0a-4295-886d-9d238a646b9a" (UID: "591b0147-7c0a-4295-886d-9d238a646b9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:03.293785 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.288935 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b79b5853-8745-436a-aeff-aada86fb4853-plugins-config-volume" pod "b79b5853-8745-436a-aeff-aada86fb4853" (UID: "b79b5853-8745-436a-aeff-aada86fb4853"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:03.357234 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.356538 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b79b5853-8745-436a-aeff-aada86fb4853-kube-api-access-lrbvj" pod "b79b5853-8745-436a-aeff-aada86fb4853" (UID: "b79b5853-8745-436a-aeff-aada86fb4853"). InnerVolumeSpecName "kube-api-access-lrbvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:03.357386 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.357325 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591b0147-7c0a-4295-886d-9d238a646b9a-kube-api-access-2jh7f" pod "591b0147-7c0a-4295-886d-9d238a646b9a" (UID: "591b0147-7c0a-4295-886d-9d238a646b9a"). InnerVolumeSpecName "kube-api-access-2jh7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:03.388914 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.388885 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jh7f\" (UniqueName: \"kubernetes.io/projected/591b0147-7c0a-4295-886d-9d238a646b9a-kube-api-access-2jh7f\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:34:03.388914 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.388912 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrbvj\" (UniqueName: \"kubernetes.io/projected/b79b5853-8745-436a-aeff-aada86fb4853-kube-api-access-lrbvj\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:34:03.388914 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.388921 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591b0147-7c0a-4295-886d-9d238a646b9a-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:34:03.389162 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.388931 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b79b5853-8745-436a-aeff-aada86fb4853-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:34:03.421332 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.421307 4603 generic.go:356] "Generic (PLEG): container finished" podID="591b0147-7c0a-4295-886d-9d238a646b9a" containerID="19a20d6097bc953ffa521639253cfb183315b904328bac6c7e403f00354d5a1d" exitCode=0 Oct 30 18:34:03.421851 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.421364 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk" Oct 30 18:34:03.421851 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.421382 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk" event={"ID":"591b0147-7c0a-4295-886d-9d238a646b9a","Type":"ContainerDied","Data":"19a20d6097bc953ffa521639253cfb183315b904328bac6c7e403f00354d5a1d"} Oct 30 18:34:03.421851 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.421429 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk" event={"ID":"591b0147-7c0a-4295-886d-9d238a646b9a","Type":"ContainerDied","Data":"17b94222a287aebc1f30809bdaa9256b06c5656cb6da290386dda57fcd9caff4"} Oct 30 18:34:03.421851 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.421446 4603 scope.go:122] "RemoveContainer" containerID="19a20d6097bc953ffa521639253cfb183315b904328bac6c7e403f00354d5a1d" Oct 30 18:34:03.423082 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.423061 4603 generic.go:356] "Generic (PLEG): container finished" podID="b79b5853-8745-436a-aeff-aada86fb4853" containerID="4f0b6dcca8452cf73c69505ae2b28df2f25391a129e8877818f12595f4dc6bcf" exitCode=0 Oct 30 18:34:03.423206 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.423107 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9" Oct 30 18:34:03.423206 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.423147 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9" event={"ID":"b79b5853-8745-436a-aeff-aada86fb4853","Type":"ContainerDied","Data":"4f0b6dcca8452cf73c69505ae2b28df2f25391a129e8877818f12595f4dc6bcf"} Oct 30 18:34:03.423351 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.423207 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9" event={"ID":"b79b5853-8745-436a-aeff-aada86fb4853","Type":"ContainerDied","Data":"c296c6af29051f4c2275cf6b7c3ad8666a7b35bd510ceac371e816765309dfdd"} Oct 30 18:34:03.432456 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.432441 4603 scope.go:122] "RemoveContainer" containerID="19a20d6097bc953ffa521639253cfb183315b904328bac6c7e403f00354d5a1d" Oct 30 18:34:03.432737 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:34:03.432720 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19a20d6097bc953ffa521639253cfb183315b904328bac6c7e403f00354d5a1d\": container with ID starting with 19a20d6097bc953ffa521639253cfb183315b904328bac6c7e403f00354d5a1d not found: ID does not exist" containerID="19a20d6097bc953ffa521639253cfb183315b904328bac6c7e403f00354d5a1d" Oct 30 18:34:03.432783 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.432745 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19a20d6097bc953ffa521639253cfb183315b904328bac6c7e403f00354d5a1d"} err="failed to get container status \"19a20d6097bc953ffa521639253cfb183315b904328bac6c7e403f00354d5a1d\": rpc error: code = NotFound desc = could not find container \"19a20d6097bc953ffa521639253cfb183315b904328bac6c7e403f00354d5a1d\": container with ID starting with 19a20d6097bc953ffa521639253cfb183315b904328bac6c7e403f00354d5a1d not found: ID does not exist" Oct 30 18:34:03.432783 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.432761 4603 scope.go:122] "RemoveContainer" containerID="4f0b6dcca8452cf73c69505ae2b28df2f25391a129e8877818f12595f4dc6bcf" Oct 30 18:34:03.441911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.441885 4603 scope.go:122] "RemoveContainer" containerID="4f0b6dcca8452cf73c69505ae2b28df2f25391a129e8877818f12595f4dc6bcf" Oct 30 18:34:03.442169 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:34:03.442153 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0b6dcca8452cf73c69505ae2b28df2f25391a129e8877818f12595f4dc6bcf\": container with ID starting with 4f0b6dcca8452cf73c69505ae2b28df2f25391a129e8877818f12595f4dc6bcf not found: ID does not exist" containerID="4f0b6dcca8452cf73c69505ae2b28df2f25391a129e8877818f12595f4dc6bcf" Oct 30 18:34:03.442224 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.442174 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0b6dcca8452cf73c69505ae2b28df2f25391a129e8877818f12595f4dc6bcf"} err="failed to get container status \"4f0b6dcca8452cf73c69505ae2b28df2f25391a129e8877818f12595f4dc6bcf\": rpc error: code = NotFound desc = could not find container \"4f0b6dcca8452cf73c69505ae2b28df2f25391a129e8877818f12595f4dc6bcf\": container with ID starting with 4f0b6dcca8452cf73c69505ae2b28df2f25391a129e8877818f12595f4dc6bcf not found: ID does not exist" Oct 30 18:34:03.474829 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.474776 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk"] Oct 30 18:34:03.479097 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.479074 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-864cf77955-x2nrk"] Oct 30 18:34:03.513041 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.513015 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9"] Oct 30 18:34:03.517850 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.517829 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-602e18-679cf796b4-7j4m9"] Oct 30 18:34:03.761426 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:03.761335 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:04.941324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:04.941287 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:05.412081 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:05.412048 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="591b0147-7c0a-4295-886d-9d238a646b9a" path="/var/lib/kubelet/pods/591b0147-7c0a-4295-886d-9d238a646b9a/volumes" Oct 30 18:34:05.412536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:05.412523 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b79b5853-8745-436a-aeff-aada86fb4853" path="/var/lib/kubelet/pods/b79b5853-8745-436a-aeff-aada86fb4853/volumes" Oct 30 18:34:06.655554 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:06.652566 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:08.194077 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:08.194040 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:09.837384 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:09.835408 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:10.274397 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:10.274348 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:10.414559 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:10.414479 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:10.579321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:10.579239 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz"] Oct 30 18:34:10.579850 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:10.579818 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" podUID="07edd101-b4e3-48b5-95fc-2168b424cfd9" containerName="istio-proxy" containerID="cri-o://ade79797f9d0448cb267c8c6469108edeebf906ca930dfb8176652d30b8d842f" gracePeriod=30 Oct 30 18:34:10.643752 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:10.643727 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:11.496489 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:11.496456 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:12.217727 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:12.217695 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:12.760220 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:12.760187 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr"] Oct 30 18:34:12.773027 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:12.772999 4603 cpu_manager.go:414] "containerMap: removing container" podUID="b79b5853-8745-436a-aeff-aada86fb4853" containerName="payload-processing" Oct 30 18:34:12.773250 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:12.773234 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b79b5853-8745-436a-aeff-aada86fb4853" containerName="payload-processing" Oct 30 18:34:12.773373 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:12.773362 4603 cpu_manager.go:414] "containerMap: removing container" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerName="istio-proxy" Oct 30 18:34:12.773454 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:12.773442 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerName="istio-proxy" Oct 30 18:34:12.773578 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:12.773565 4603 cpu_manager.go:414] "containerMap: removing container" podUID="a30517af-31fe-458a-8b17-40bc5ab61274" containerName="istio-proxy" Oct 30 18:34:12.773669 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:12.773658 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a30517af-31fe-458a-8b17-40bc5ab61274" containerName="istio-proxy" Oct 30 18:34:12.773819 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:12.773808 4603 cpu_manager.go:414] "containerMap: removing container" podUID="591b0147-7c0a-4295-886d-9d238a646b9a" containerName="payload-pre-processing" Oct 30 18:34:12.773894 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:12.773883 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="591b0147-7c0a-4295-886d-9d238a646b9a" containerName="payload-pre-processing" Oct 30 18:34:12.773977 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:12.773968 4603 cpu_manager.go:414] "containerMap: removing container" podUID="a065f551-ec87-4081-bc18-27d0dd4234e3" containerName="istio-proxy" Oct 30 18:34:12.774063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:12.774053 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a065f551-ec87-4081-bc18-27d0dd4234e3" containerName="istio-proxy" Oct 30 18:34:12.774369 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:12.774349 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerName="istio-proxy" Oct 30 18:34:12.774577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:12.774562 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="a30517af-31fe-458a-8b17-40bc5ab61274" containerName="istio-proxy" Oct 30 18:34:12.774698 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:12.774686 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerName="istio-proxy" Oct 30 18:34:12.774791 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:12.774781 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="591b0147-7c0a-4295-886d-9d238a646b9a" containerName="payload-pre-processing" Oct 30 18:34:12.774873 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:12.774863 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="a065f551-ec87-4081-bc18-27d0dd4234e3" containerName="istio-proxy" Oct 30 18:34:12.774960 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:12.774950 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="b79b5853-8745-436a-aeff-aada86fb4853" containerName="payload-processing" Oct 30 18:34:12.797532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:12.794564 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.006824 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.006616 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-ac422f-openshift-default-dockercfg-w59vs\"" Oct 30 18:34:13.070527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.019431 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:13.097350 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.097081 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr"] Oct 30 18:34:13.159894 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.159843 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-token\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.160111 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.160074 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1892e5-3097-4ce7-8138-caceb31225b8-istiod-ca-cert\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.160207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.160186 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-workload-socket\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.160276 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.160238 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-podinfo\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.160342 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.160274 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-credential-socket\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.160342 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.160300 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-data\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.160342 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.160330 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f62m6\" (UniqueName: \"kubernetes.io/projected/ec1892e5-3097-4ce7-8138-caceb31225b8-kube-api-access-f62m6\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.160582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.160390 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-envoy\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.160582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.160425 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-workload-certs\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.230103 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.229762 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:13.261900 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.261861 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-credential-socket\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.262116 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.261915 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-data\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.262116 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.261945 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f62m6\" (UniqueName: \"kubernetes.io/projected/ec1892e5-3097-4ce7-8138-caceb31225b8-kube-api-access-f62m6\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.262116 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.262004 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-envoy\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.262116 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.262049 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-workload-certs\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.262360 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.262191 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-token\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.262360 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.262302 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-credential-socket\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.262745 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.262713 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1892e5-3097-4ce7-8138-caceb31225b8-istiod-ca-cert\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.262897 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.262815 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-workload-socket\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.262897 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.262851 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-podinfo\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.262897 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.262884 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-data\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.263851 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.262928 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-workload-certs\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.263851 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.263213 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-workload-socket\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.263851 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.263595 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1892e5-3097-4ce7-8138-caceb31225b8-istiod-ca-cert\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.265987 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.265952 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-envoy\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.266407 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.266379 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-podinfo\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.286195 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.286160 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f62m6\" (UniqueName: \"kubernetes.io/projected/ec1892e5-3097-4ce7-8138-caceb31225b8-kube-api-access-f62m6\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.286474 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.286456 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-token\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.467214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.467130 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:13.696317 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.696286 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:13.798298 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.798274 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr"] Oct 30 18:34:13.798697 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:34:13.798549 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1892e5_3097_4ce7_8138_caceb31225b8.slice/crio-f29a41298a361cab25093cd80c1754972a5bef66e7d5e4e06b03cfda3636ac63 WatchSource:0}: Error finding container f29a41298a361cab25093cd80c1754972a5bef66e7d5e4e06b03cfda3636ac63: Status 404 returned error can't find the container with id f29a41298a361cab25093cd80c1754972a5bef66e7d5e4e06b03cfda3636ac63 Oct 30 18:34:13.801568 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.801491 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:34:13.801685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.801625 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:34:13.801745 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:13.801695 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:34:14.506491 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:14.506457 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" event={"ID":"ec1892e5-3097-4ce7-8138-caceb31225b8","Type":"ContainerStarted","Data":"90c73c94b47ccb54723aa8dbb9d8b36ad0c10d75451978cd5531774413ab2854"} Oct 30 18:34:14.506701 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:14.506547 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" event={"ID":"ec1892e5-3097-4ce7-8138-caceb31225b8","Type":"ContainerStarted","Data":"f29a41298a361cab25093cd80c1754972a5bef66e7d5e4e06b03cfda3636ac63"} Oct 30 18:34:14.565808 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:14.565762 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" podStartSLOduration=2.565744589 podStartE2EDuration="2.565744589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:14.562107963 +0000 UTC m=+2515.277677837" watchObservedRunningTime="2026-10-30 18:34:14.565744589 +0000 UTC m=+2515.281314464" Oct 30 18:34:15.355309 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:15.355272 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:15.471534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:15.469912 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:15.491954 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:15.491918 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:15.514677 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:15.514641 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:15.515727 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:15.515700 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:34:15.882874 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:15.882817 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:34:16.044874 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.044840 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-podinfo\") pod \"07edd101-b4e3-48b5-95fc-2168b424cfd9\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " Oct 30 18:34:16.045060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.045008 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07edd101-b4e3-48b5-95fc-2168b424cfd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07edd101-b4e3-48b5-95fc-2168b424cfd9-istiod-ca-cert\") pod \"07edd101-b4e3-48b5-95fc-2168b424cfd9\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " Oct 30 18:34:16.045135 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.045123 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-data\") pod \"07edd101-b4e3-48b5-95fc-2168b424cfd9\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " Oct 30 18:34:16.045195 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.045182 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-workload-certs\") pod \"07edd101-b4e3-48b5-95fc-2168b424cfd9\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " Oct 30 18:34:16.045251 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.045216 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07edd101-b4e3-48b5-95fc-2168b424cfd9-kube-api-access-dlkbr\" (UniqueName: \"kubernetes.io/projected/07edd101-b4e3-48b5-95fc-2168b424cfd9-kube-api-access-dlkbr\") pod \"07edd101-b4e3-48b5-95fc-2168b424cfd9\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " Oct 30 18:34:16.045251 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.045240 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-credential-socket\") pod \"07edd101-b4e3-48b5-95fc-2168b424cfd9\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " Oct 30 18:34:16.045367 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.045313 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-token\" (UniqueName: \"kubernetes.io/projected/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-token\") pod \"07edd101-b4e3-48b5-95fc-2168b424cfd9\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " Oct 30 18:34:16.045428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.045389 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-workload-socket\") pod \"07edd101-b4e3-48b5-95fc-2168b424cfd9\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " Oct 30 18:34:16.045488 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.045418 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-data" pod "07edd101-b4e3-48b5-95fc-2168b424cfd9" (UID: "07edd101-b4e3-48b5-95fc-2168b424cfd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:16.045488 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.045439 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-envoy\") pod \"07edd101-b4e3-48b5-95fc-2168b424cfd9\" (UID: \"07edd101-b4e3-48b5-95fc-2168b424cfd9\") " Oct 30 18:34:16.045488 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.045475 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07edd101-b4e3-48b5-95fc-2168b424cfd9-istiod-ca-cert" pod "07edd101-b4e3-48b5-95fc-2168b424cfd9" (UID: "07edd101-b4e3-48b5-95fc-2168b424cfd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:16.045800 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.045763 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-credential-socket" pod "07edd101-b4e3-48b5-95fc-2168b424cfd9" (UID: "07edd101-b4e3-48b5-95fc-2168b424cfd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:16.046064 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.045984 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-workload-certs" pod "07edd101-b4e3-48b5-95fc-2168b424cfd9" (UID: "07edd101-b4e3-48b5-95fc-2168b424cfd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:16.046276 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.046173 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-workload-socket" pod "07edd101-b4e3-48b5-95fc-2168b424cfd9" (UID: "07edd101-b4e3-48b5-95fc-2168b424cfd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:16.046438 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.046416 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07edd101-b4e3-48b5-95fc-2168b424cfd9-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:34:16.046551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.046445 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:34:16.046551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.046460 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:34:16.046551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.046474 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:34:16.046744 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.046599 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:34:16.047324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.047295 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-podinfo" pod "07edd101-b4e3-48b5-95fc-2168b424cfd9" (UID: "07edd101-b4e3-48b5-95fc-2168b424cfd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:34:16.047709 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.047684 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07edd101-b4e3-48b5-95fc-2168b424cfd9-kube-api-access-dlkbr" pod "07edd101-b4e3-48b5-95fc-2168b424cfd9" (UID: "07edd101-b4e3-48b5-95fc-2168b424cfd9"). InnerVolumeSpecName "kube-api-access-dlkbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:16.047822 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.047738 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-token" pod "07edd101-b4e3-48b5-95fc-2168b424cfd9" (UID: "07edd101-b4e3-48b5-95fc-2168b424cfd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:16.048324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.048303 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-envoy" pod "07edd101-b4e3-48b5-95fc-2168b424cfd9" (UID: "07edd101-b4e3-48b5-95fc-2168b424cfd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:16.148545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.148439 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:34:16.148545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.148469 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:34:16.148545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.148478 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07edd101-b4e3-48b5-95fc-2168b424cfd9-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:34:16.148545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.148488 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlkbr\" (UniqueName: \"kubernetes.io/projected/07edd101-b4e3-48b5-95fc-2168b424cfd9-kube-api-access-dlkbr\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:34:16.522681 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.522649 4603 generic.go:356] "Generic (PLEG): container finished" podID="07edd101-b4e3-48b5-95fc-2168b424cfd9" containerID="ade79797f9d0448cb267c8c6469108edeebf906ca930dfb8176652d30b8d842f" exitCode=0 Oct 30 18:34:16.523115 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.522701 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" Oct 30 18:34:16.523115 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.522723 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" event={"ID":"07edd101-b4e3-48b5-95fc-2168b424cfd9","Type":"ContainerDied","Data":"ade79797f9d0448cb267c8c6469108edeebf906ca930dfb8176652d30b8d842f"} Oct 30 18:34:16.523115 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.522768 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz" event={"ID":"07edd101-b4e3-48b5-95fc-2168b424cfd9","Type":"ContainerDied","Data":"57bb1750797f648a322a1b07a51fb24a3b451e0b65a1fadca362628892c3bc40"} Oct 30 18:34:16.523115 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.522789 4603 scope.go:122] "RemoveContainer" containerID="ade79797f9d0448cb267c8c6469108edeebf906ca930dfb8176652d30b8d842f" Oct 30 18:34:16.533851 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.533833 4603 scope.go:122] "RemoveContainer" containerID="71a667c6a43e514f3e9d7f5cddf6ed3c161e7ffbcd10947dd4efecaca8a18773" Oct 30 18:34:16.545677 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.545656 4603 scope.go:122] "RemoveContainer" containerID="ade79797f9d0448cb267c8c6469108edeebf906ca930dfb8176652d30b8d842f" Oct 30 18:34:16.545972 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:34:16.545944 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ade79797f9d0448cb267c8c6469108edeebf906ca930dfb8176652d30b8d842f\": container with ID starting with ade79797f9d0448cb267c8c6469108edeebf906ca930dfb8176652d30b8d842f not found: ID does not exist" containerID="ade79797f9d0448cb267c8c6469108edeebf906ca930dfb8176652d30b8d842f" Oct 30 18:34:16.546025 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.545979 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ade79797f9d0448cb267c8c6469108edeebf906ca930dfb8176652d30b8d842f"} err="failed to get container status \"ade79797f9d0448cb267c8c6469108edeebf906ca930dfb8176652d30b8d842f\": rpc error: code = NotFound desc = could not find container \"ade79797f9d0448cb267c8c6469108edeebf906ca930dfb8176652d30b8d842f\": container with ID starting with ade79797f9d0448cb267c8c6469108edeebf906ca930dfb8176652d30b8d842f not found: ID does not exist" Oct 30 18:34:16.546025 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.545996 4603 scope.go:122] "RemoveContainer" containerID="71a667c6a43e514f3e9d7f5cddf6ed3c161e7ffbcd10947dd4efecaca8a18773" Oct 30 18:34:16.546240 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:34:16.546220 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71a667c6a43e514f3e9d7f5cddf6ed3c161e7ffbcd10947dd4efecaca8a18773\": container with ID starting with 71a667c6a43e514f3e9d7f5cddf6ed3c161e7ffbcd10947dd4efecaca8a18773 not found: ID does not exist" containerID="71a667c6a43e514f3e9d7f5cddf6ed3c161e7ffbcd10947dd4efecaca8a18773" Oct 30 18:34:16.546297 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.546250 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a667c6a43e514f3e9d7f5cddf6ed3c161e7ffbcd10947dd4efecaca8a18773"} err="failed to get container status \"71a667c6a43e514f3e9d7f5cddf6ed3c161e7ffbcd10947dd4efecaca8a18773\": rpc error: code = NotFound desc = could not find container \"71a667c6a43e514f3e9d7f5cddf6ed3c161e7ffbcd10947dd4efecaca8a18773\": container with ID starting with 71a667c6a43e514f3e9d7f5cddf6ed3c161e7ffbcd10947dd4efecaca8a18773 not found: ID does not exist" Oct 30 18:34:16.576128 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.576107 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz"] Oct 30 18:34:16.588237 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.588219 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-602e18-openshift-default-77b95fd77-7q2jz"] Oct 30 18:34:16.952635 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:16.952550 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:17.412266 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:17.412238 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07edd101-b4e3-48b5-95fc-2168b424cfd9" path="/var/lib/kubelet/pods/07edd101-b4e3-48b5-95fc-2168b424cfd9/volumes" Oct 30 18:34:17.891220 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:17.891191 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:18.609935 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:18.602323 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:18.750340 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:18.750303 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:20.246195 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:20.246162 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:21.930686 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:34:21.930653 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:34:21.948631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:21.948601 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:23.641574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:23.641537 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:23.850126 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:23.850087 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:25.405472 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:25.405435 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:27.143435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:27.143398 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:28.651191 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:28.651145 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:30.454071 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:30.454026 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:31.094883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:31.094855 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:32.259968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:32.259932 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:34.072055 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:34.072016 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:35.782539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:35.780351 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:37.112700 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:37.112665 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:37.261824 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:37.261791 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:37.618219 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:37.618188 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:38.832630 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:38.832593 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:38.978415 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:38.978384 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw"] Oct 30 18:34:38.979086 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:38.979070 4603 cpu_manager.go:414] "containerMap: removing container" podUID="07edd101-b4e3-48b5-95fc-2168b424cfd9" containerName="istio-proxy" Oct 30 18:34:38.979144 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:38.979088 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07edd101-b4e3-48b5-95fc-2168b424cfd9" containerName="istio-proxy" Oct 30 18:34:38.979144 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:38.979100 4603 cpu_manager.go:414] "containerMap: removing container" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerName="istio-proxy" Oct 30 18:34:38.979144 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:38.979105 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f4e2fe-260f-49bd-9c9f-dc73eff6c5af" containerName="istio-proxy" Oct 30 18:34:38.979144 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:38.979121 4603 cpu_manager.go:414] "containerMap: removing container" podUID="07edd101-b4e3-48b5-95fc-2168b424cfd9" containerName="istio-proxy" Oct 30 18:34:38.979144 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:38.979126 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07edd101-b4e3-48b5-95fc-2168b424cfd9" containerName="istio-proxy" Oct 30 18:34:38.979305 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:38.979255 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="07edd101-b4e3-48b5-95fc-2168b424cfd9" containerName="istio-proxy" Oct 30 18:34:38.979305 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:38.979265 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="07edd101-b4e3-48b5-95fc-2168b424cfd9" containerName="istio-proxy" Oct 30 18:34:38.984078 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:38.984060 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw" Oct 30 18:34:39.008102 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:39.008070 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:39.019000 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:39.012597 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-ac422f\"" Oct 30 18:34:39.019000 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:39.012625 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-ac422f-dockercfg-nx2hv\"" Oct 30 18:34:39.019000 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:39.018937 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw"] Oct 30 18:34:39.166581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:39.161527 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9f4fc8-48f9-4e12-afa4-fecea8575b5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw\" (UID: \"4d9f4fc8-48f9-4e12-afa4-fecea8575b5c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw" Oct 30 18:34:39.166581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:39.161739 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5vqw\" (UniqueName: \"kubernetes.io/projected/4d9f4fc8-48f9-4e12-afa4-fecea8575b5c-kube-api-access-f5vqw\") pod \"payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw\" (UID: \"4d9f4fc8-48f9-4e12-afa4-fecea8575b5c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw" Oct 30 18:34:39.203527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:39.199415 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r"] Oct 30 18:34:39.321531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:39.263952 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9f4fc8-48f9-4e12-afa4-fecea8575b5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw\" (UID: \"4d9f4fc8-48f9-4e12-afa4-fecea8575b5c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw" Oct 30 18:34:39.321531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:39.264149 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5vqw\" (UniqueName: \"kubernetes.io/projected/4d9f4fc8-48f9-4e12-afa4-fecea8575b5c-kube-api-access-f5vqw\") pod \"payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw\" (UID: \"4d9f4fc8-48f9-4e12-afa4-fecea8575b5c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw" Oct 30 18:34:39.321531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:39.265319 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9f4fc8-48f9-4e12-afa4-fecea8575b5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw\" (UID: \"4d9f4fc8-48f9-4e12-afa4-fecea8575b5c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw" Oct 30 18:34:39.321531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:39.276173 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r" Oct 30 18:34:39.321531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:39.292416 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r"] Oct 30 18:34:39.377532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:39.377424 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d52a97-9fde-4a62-95ab-b6fa18c65db9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r\" (UID: \"f7d52a97-9fde-4a62-95ab-b6fa18c65db9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r" Oct 30 18:34:39.377736 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:39.377599 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkd6s\" (UniqueName: \"kubernetes.io/projected/f7d52a97-9fde-4a62-95ab-b6fa18c65db9-kube-api-access-kkd6s\") pod \"payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r\" (UID: \"f7d52a97-9fde-4a62-95ab-b6fa18c65db9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r" Oct 30 18:34:39.415526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:39.410305 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:39.573763 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:39.571920 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkd6s\" (UniqueName: \"kubernetes.io/projected/f7d52a97-9fde-4a62-95ab-b6fa18c65db9-kube-api-access-kkd6s\") pod \"payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r\" (UID: \"f7d52a97-9fde-4a62-95ab-b6fa18c65db9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r" Oct 30 18:34:39.573763 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:39.572425 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d52a97-9fde-4a62-95ab-b6fa18c65db9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r\" (UID: \"f7d52a97-9fde-4a62-95ab-b6fa18c65db9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r" Oct 30 18:34:39.574184 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:39.574155 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d52a97-9fde-4a62-95ab-b6fa18c65db9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r\" (UID: \"f7d52a97-9fde-4a62-95ab-b6fa18c65db9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r" Oct 30 18:34:39.608910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:39.608827 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5vqw\" (UniqueName: \"kubernetes.io/projected/4d9f4fc8-48f9-4e12-afa4-fecea8575b5c-kube-api-access-f5vqw\") pod \"payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw\" (UID: \"4d9f4fc8-48f9-4e12-afa4-fecea8575b5c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw" Oct 30 18:34:39.612036 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:39.612007 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw" Oct 30 18:34:39.636809 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:39.636783 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkd6s\" (UniqueName: \"kubernetes.io/projected/f7d52a97-9fde-4a62-95ab-b6fa18c65db9-kube-api-access-kkd6s\") pod \"payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r\" (UID: \"f7d52a97-9fde-4a62-95ab-b6fa18c65db9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r" Oct 30 18:34:39.914861 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:39.914793 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r" Oct 30 18:34:39.943457 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:34:39.943429 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9f4fc8_48f9_4e12_afa4_fecea8575b5c.slice/crio-e77aa3ed21e827dde349f3ab4dca7f5cb536d665786965a2e6df8d2c3bd47090 WatchSource:0}: Error finding container e77aa3ed21e827dde349f3ab4dca7f5cb536d665786965a2e6df8d2c3bd47090: Status 404 returned error can't find the container with id e77aa3ed21e827dde349f3ab4dca7f5cb536d665786965a2e6df8d2c3bd47090 Oct 30 18:34:39.943904 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:39.943876 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw"] Oct 30 18:34:40.227942 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:40.227911 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r"] Oct 30 18:34:40.228820 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:34:40.228795 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d52a97_9fde_4a62_95ab_b6fa18c65db9.slice/crio-d4288c0ddfa79000b580a6461af331da97dd5926c73725d6c95f9fc9b574fbf6 WatchSource:0}: Error finding container d4288c0ddfa79000b580a6461af331da97dd5926c73725d6c95f9fc9b574fbf6: Status 404 returned error can't find the container with id d4288c0ddfa79000b580a6461af331da97dd5926c73725d6c95f9fc9b574fbf6 Oct 30 18:34:40.780120 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:40.780085 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw" event={"ID":"4d9f4fc8-48f9-4e12-afa4-fecea8575b5c","Type":"ContainerStarted","Data":"32bdcf8d25dd9ebc0df48beb079f3030d10113d3c0ef604e737cdcec61987f6d"} Oct 30 18:34:40.780285 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:40.780148 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw" Oct 30 18:34:40.780285 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:40.780175 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw" event={"ID":"4d9f4fc8-48f9-4e12-afa4-fecea8575b5c","Type":"ContainerStarted","Data":"e77aa3ed21e827dde349f3ab4dca7f5cb536d665786965a2e6df8d2c3bd47090"} Oct 30 18:34:40.781772 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:40.781742 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r" event={"ID":"f7d52a97-9fde-4a62-95ab-b6fa18c65db9","Type":"ContainerStarted","Data":"e8f13b5998bd6d3ce2137a2325229c80dcb3337b6d1480a2fdc3174bb532a65c"} Oct 30 18:34:40.781915 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:40.781789 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r" event={"ID":"f7d52a97-9fde-4a62-95ab-b6fa18c65db9","Type":"ContainerStarted","Data":"d4288c0ddfa79000b580a6461af331da97dd5926c73725d6c95f9fc9b574fbf6"} Oct 30 18:34:40.781915 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:40.781816 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r" Oct 30 18:34:40.824488 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:40.824432 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw" podStartSLOduration=2.824416823 podStartE2EDuration="2.824416823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:40.820456969 +0000 UTC m=+2541.536026843" watchObservedRunningTime="2026-10-30 18:34:40.824416823 +0000 UTC m=+2541.539986699" Oct 30 18:34:40.862107 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:40.862056 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r" podStartSLOduration=1.862039671 podStartE2EDuration="1.862039671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:40.860485201 +0000 UTC m=+2541.576055075" watchObservedRunningTime="2026-10-30 18:34:40.862039671 +0000 UTC m=+2541.577609526" Oct 30 18:34:41.053867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:41.053786 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:42.649606 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:42.649569 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:44.326336 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:44.326301 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:46.069550 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:46.060731 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:47.634496 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:47.634461 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:49.319083 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:49.319040 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:50.914018 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:50.913984 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:51.791480 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:51.791443 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw" Oct 30 18:34:51.791887 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:51.791867 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r" Oct 30 18:34:51.954349 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:34:51.954306 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:34:52.410955 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:52.410907 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:52.779114 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:52.779079 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:54.552768 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:54.552734 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:56.217631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:56.211452 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:57.760530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:57.757157 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:59.611571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:59.597441 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:59.756929 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:34:59.756876 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:01.135548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:01.135430 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:01.323528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:01.314271 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:02.958313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:02.955180 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:04.626566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:04.625821 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:06.262153 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:06.262114 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:07.959462 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:07.959423 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:09.485398 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:09.485365 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:09.675697 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:09.675667 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:09.753689 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:09.753604 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7322-eef857"] Oct 30 18:35:09.779287 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:09.779256 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7322-eef857"] Oct 30 18:35:09.779647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:09.779597 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-eef857" Oct 30 18:35:09.795226 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:09.795170 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 18:35:09.825951 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:09.824909 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 18:35:09.825951 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:09.825003 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vdx7j\"" Oct 30 18:35:09.895356 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:09.895312 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mhpb\" (UniqueName: \"kubernetes.io/projected/8ef94a04-c107-430e-9479-ae7aa6e5d5dc-kube-api-access-8mhpb\") pod \"test-curl-7322-eef857\" (UID: \"8ef94a04-c107-430e-9479-ae7aa6e5d5dc\") " pod="opendatahub/test-curl-7322-eef857" Oct 30 18:35:09.997264 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:09.997229 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mhpb\" (UniqueName: \"kubernetes.io/projected/8ef94a04-c107-430e-9479-ae7aa6e5d5dc-kube-api-access-8mhpb\") pod \"test-curl-7322-eef857\" (UID: \"8ef94a04-c107-430e-9479-ae7aa6e5d5dc\") " pod="opendatahub/test-curl-7322-eef857" Oct 30 18:35:10.020164 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:10.020140 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mhpb\" (UniqueName: \"kubernetes.io/projected/8ef94a04-c107-430e-9479-ae7aa6e5d5dc-kube-api-access-8mhpb\") pod \"test-curl-7322-eef857\" (UID: \"8ef94a04-c107-430e-9479-ae7aa6e5d5dc\") " pod="opendatahub/test-curl-7322-eef857" Oct 30 18:35:10.200197 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:10.200165 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-eef857" Oct 30 18:35:10.506210 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:35:10.506175 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ef94a04_c107_430e_9479_ae7aa6e5d5dc.slice/crio-4c8e8486d977bb635551b4076c22360099d7eb994ae37cd7c2eaf05a51d037a4 WatchSource:0}: Error finding container 4c8e8486d977bb635551b4076c22360099d7eb994ae37cd7c2eaf05a51d037a4: Status 404 returned error can't find the container with id 4c8e8486d977bb635551b4076c22360099d7eb994ae37cd7c2eaf05a51d037a4 Oct 30 18:35:10.506606 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:10.506336 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7322-eef857"] Oct 30 18:35:11.088096 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:11.088061 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7322-eef857" event={"ID":"8ef94a04-c107-430e-9479-ae7aa6e5d5dc","Type":"ContainerStarted","Data":"d3c98166e5db461d7ec670b550eb17dbc72aa8c32ec074a3289d9374e50f0a20"} Oct 30 18:35:11.088295 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:11.088117 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7322-eef857" event={"ID":"8ef94a04-c107-430e-9479-ae7aa6e5d5dc","Type":"ContainerStarted","Data":"4c8e8486d977bb635551b4076c22360099d7eb994ae37cd7c2eaf05a51d037a4"} Oct 30 18:35:11.129180 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:11.129133 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7322-eef857" podStartSLOduration=2.129119565 podStartE2EDuration="2.129119565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:11.125454194 +0000 UTC m=+2571.841024067" watchObservedRunningTime="2026-10-30 18:35:11.129119565 +0000 UTC m=+2571.844689439" Oct 30 18:35:11.434307 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:11.434217 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:13.124531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:13.098569 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:14.781688 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:14.781654 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:16.442218 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:16.442175 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:17.998640 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:17.998603 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:18.191791 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:18.191751 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:18.695833 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:18.695785 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:19.742057 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:19.742003 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:21.334322 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:21.334279 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:21.933153 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:35:21.933119 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:35:23.033018 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:23.032967 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:24.605528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:24.604526 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:26.128117 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:26.128084 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:26.286429 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:26.286397 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7322-eef857"] Oct 30 18:35:26.286689 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:26.286644 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7322-eef857" podUID="8ef94a04-c107-430e-9479-ae7aa6e5d5dc" containerName="test-curl-7322-eef857" containerID="cri-o://d3c98166e5db461d7ec670b550eb17dbc72aa8c32ec074a3289d9374e50f0a20" gracePeriod=2 Oct 30 18:35:26.293273 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:26.293248 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7322-eef857"] Oct 30 18:35:26.300016 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:35:26.299980 4603 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-eef857\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="8ef94a04-c107-430e-9479-ae7aa6e5d5dc" pod="opendatahub/test-curl-7322-eef857" Oct 30 18:35:26.461758 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:26.461686 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7322-0bf4a7"] Oct 30 18:35:26.462372 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:26.462358 4603 cpu_manager.go:414] "containerMap: removing container" podUID="8ef94a04-c107-430e-9479-ae7aa6e5d5dc" containerName="test-curl-7322-eef857" Oct 30 18:35:26.462421 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:26.462374 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ef94a04-c107-430e-9479-ae7aa6e5d5dc" containerName="test-curl-7322-eef857" Oct 30 18:35:26.462536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:26.462492 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ef94a04-c107-430e-9479-ae7aa6e5d5dc" containerName="test-curl-7322-eef857" Oct 30 18:35:26.466058 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:26.466035 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-0bf4a7" Oct 30 18:35:26.484845 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:26.484821 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7322-0bf4a7"] Oct 30 18:35:26.515149 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:35:26.515122 4603 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-eef857\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="8ef94a04-c107-430e-9479-ae7aa6e5d5dc" pod="opendatahub/test-curl-7322-eef857" Oct 30 18:35:26.583061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:26.583032 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbflf\" (UniqueName: \"kubernetes.io/projected/f0c2b609-d1ed-4ca7-9a35-e63146546568-kube-api-access-qbflf\") pod \"test-curl-7322-0bf4a7\" (UID: \"f0c2b609-d1ed-4ca7-9a35-e63146546568\") " pod="opendatahub/test-curl-7322-0bf4a7" Oct 30 18:35:26.684136 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:26.684104 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbflf\" (UniqueName: \"kubernetes.io/projected/f0c2b609-d1ed-4ca7-9a35-e63146546568-kube-api-access-qbflf\") pod \"test-curl-7322-0bf4a7\" (UID: \"f0c2b609-d1ed-4ca7-9a35-e63146546568\") " pod="opendatahub/test-curl-7322-0bf4a7" Oct 30 18:35:26.707309 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:26.707279 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbflf\" (UniqueName: \"kubernetes.io/projected/f0c2b609-d1ed-4ca7-9a35-e63146546568-kube-api-access-qbflf\") pod \"test-curl-7322-0bf4a7\" (UID: \"f0c2b609-d1ed-4ca7-9a35-e63146546568\") " pod="opendatahub/test-curl-7322-0bf4a7" Oct 30 18:35:26.778308 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:26.778286 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-0bf4a7" Oct 30 18:35:27.087340 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:27.087306 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7322-0bf4a7"] Oct 30 18:35:27.087735 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:35:27.087705 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0c2b609_d1ed_4ca7_9a35_e63146546568.slice/crio-78c08aabf8416465cd3291647e5d9e799f9ae5cf6e82401c54a995a096bd8620 WatchSource:0}: Error finding container 78c08aabf8416465cd3291647e5d9e799f9ae5cf6e82401c54a995a096bd8620: Status 404 returned error can't find the container with id 78c08aabf8416465cd3291647e5d9e799f9ae5cf6e82401c54a995a096bd8620 Oct 30 18:35:27.245390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:27.245361 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7322-0bf4a7" event={"ID":"f0c2b609-d1ed-4ca7-9a35-e63146546568","Type":"ContainerStarted","Data":"cad78504e0695ed56c4aa5bf5e23193a4271ef345bde80325189373a51002709"} Oct 30 18:35:27.245725 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:27.245397 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7322-0bf4a7" event={"ID":"f0c2b609-d1ed-4ca7-9a35-e63146546568","Type":"ContainerStarted","Data":"78c08aabf8416465cd3291647e5d9e799f9ae5cf6e82401c54a995a096bd8620"} Oct 30 18:35:27.258683 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:35:27.258654 4603 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-eef857\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="8ef94a04-c107-430e-9479-ae7aa6e5d5dc" pod="opendatahub/test-curl-7322-eef857" Oct 30 18:35:27.298208 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:27.298156 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7322-0bf4a7" podStartSLOduration=1.298143745 podStartE2EDuration="1.298143745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:27.295542879 +0000 UTC m=+2588.011112747" watchObservedRunningTime="2026-10-30 18:35:27.298143745 +0000 UTC m=+2588.013713619" Oct 30 18:35:27.864129 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:27.864087 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:28.529960 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:28.529936 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-eef857" Oct 30 18:35:28.626565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:28.626464 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ef94a04-c107-430e-9479-ae7aa6e5d5dc-kube-api-access-8mhpb\" (UniqueName: \"kubernetes.io/projected/8ef94a04-c107-430e-9479-ae7aa6e5d5dc-kube-api-access-8mhpb\") pod \"8ef94a04-c107-430e-9479-ae7aa6e5d5dc\" (UID: \"8ef94a04-c107-430e-9479-ae7aa6e5d5dc\") " Oct 30 18:35:28.628664 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:28.628633 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef94a04-c107-430e-9479-ae7aa6e5d5dc-kube-api-access-8mhpb" pod "8ef94a04-c107-430e-9479-ae7aa6e5d5dc" (UID: "8ef94a04-c107-430e-9479-ae7aa6e5d5dc"). InnerVolumeSpecName "kube-api-access-8mhpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:35:28.728277 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:28.728247 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mhpb\" (UniqueName: \"kubernetes.io/projected/8ef94a04-c107-430e-9479-ae7aa6e5d5dc-kube-api-access-8mhpb\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:35:29.260657 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:29.260616 4603 generic.go:356] "Generic (PLEG): container finished" podID="8ef94a04-c107-430e-9479-ae7aa6e5d5dc" containerID="d3c98166e5db461d7ec670b550eb17dbc72aa8c32ec074a3289d9374e50f0a20" exitCode=137 Oct 30 18:35:29.260829 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:29.260662 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-eef857" Oct 30 18:35:29.260829 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:29.260729 4603 scope.go:122] "RemoveContainer" containerID="d3c98166e5db461d7ec670b550eb17dbc72aa8c32ec074a3289d9374e50f0a20" Oct 30 18:35:29.270806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:29.270791 4603 scope.go:122] "RemoveContainer" containerID="d3c98166e5db461d7ec670b550eb17dbc72aa8c32ec074a3289d9374e50f0a20" Oct 30 18:35:29.271038 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:35:29.271016 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3c98166e5db461d7ec670b550eb17dbc72aa8c32ec074a3289d9374e50f0a20\": container with ID starting with d3c98166e5db461d7ec670b550eb17dbc72aa8c32ec074a3289d9374e50f0a20 not found: ID does not exist" containerID="d3c98166e5db461d7ec670b550eb17dbc72aa8c32ec074a3289d9374e50f0a20" Oct 30 18:35:29.271092 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:29.271050 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c98166e5db461d7ec670b550eb17dbc72aa8c32ec074a3289d9374e50f0a20"} err="failed to get container status \"d3c98166e5db461d7ec670b550eb17dbc72aa8c32ec074a3289d9374e50f0a20\": rpc error: code = NotFound desc = could not find container \"d3c98166e5db461d7ec670b550eb17dbc72aa8c32ec074a3289d9374e50f0a20\": container with ID starting with d3c98166e5db461d7ec670b550eb17dbc72aa8c32ec074a3289d9374e50f0a20 not found: ID does not exist" Oct 30 18:35:29.282268 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:35:29.282244 4603 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-eef857\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="8ef94a04-c107-430e-9479-ae7aa6e5d5dc" pod="opendatahub/test-curl-7322-eef857" Oct 30 18:35:29.411643 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:29.411618 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ef94a04-c107-430e-9479-ae7aa6e5d5dc" path="/var/lib/kubelet/pods/8ef94a04-c107-430e-9479-ae7aa6e5d5dc/volumes" Oct 30 18:35:29.420141 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:35:29.420117 4603 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-eef857\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="8ef94a04-c107-430e-9479-ae7aa6e5d5dc" pod="opendatahub/test-curl-7322-eef857" Oct 30 18:35:29.506235 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:29.506197 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:31.202932 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:31.202898 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:31.861105 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:31.861066 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:35:32.881539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:32.875765 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:34.899594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:34.894199 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:35.506529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:35.505614 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Oct 30 18:35:35.656649 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:35.656587 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:35:35.675644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:35.675595 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:35.696689 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:35.696640 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:35.904527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:35.896708 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:35.995724 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:35.993592 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:35.995724 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:35.994004 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": context deadline exceeded" Oct 30 18:35:36.403609 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:36.403177 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:36.418530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:36.404621 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:36.418530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:36.404914 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:35:36.419194 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:36.419154 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:36.660034 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:36.659941 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:35:36.660841 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:36.660819 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:35:36.683587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:36.683553 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:36.883530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:36.782998 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": context deadline exceeded" Oct 30 18:35:36.883530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:36.856890 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:35:36.883530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:36.872936 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:35:36.883530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:36.858210 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:36.909021 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:36.908975 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:36.910379 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:36.910312 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:35:36.931700 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:36.916861 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:35:37.021530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:36.997587 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:37.021530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:36.998768 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:35:37.021530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:37.001057 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:35:37.021530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:37.001115 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:35:37.122600 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:37.121729 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:37.222126 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:37.222029 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:35:37.354538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:37.345405 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:37.446530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:37.441305 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:37.446530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:37.442486 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:37.477599 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:37.454685 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:35:37.531870 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:37.531840 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:35:37.532199 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:37.532184 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:35:37.532379 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:37.532365 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:35:37.876541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:37.875948 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:37.890528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:37.878137 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:37.923620 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:37.923593 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:35:37.943576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:37.924785 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:38.030207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:38.005590 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:38.030207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:37.925335 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:35:38.030207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:38.006685 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": context deadline exceeded" Oct 30 18:35:38.030207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:37.991621 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:38.214581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:38.214450 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:38.629545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:38.296400 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:38.629545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:38.479325 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:38.629545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:38.479421 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:38.629545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:38.481591 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:38.716383 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:38.716354 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:35:38.734456 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:38.650517 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 30 18:35:38.763937 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:38.763880 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:38.918938 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:38.918813 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:38.983680 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:38.983627 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:39.004311 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:39.004270 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:39.044106 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:39.044065 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:39.045705 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:39.044813 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:35:39.045705 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:39.045101 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:39.045705 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:39.045335 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": context deadline exceeded" Oct 30 18:35:39.045705 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:39.045640 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:35:39.612116 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:39.612074 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:39.616423 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:39.616373 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:35:39.943628 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:39.616724 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:39.943628 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:39.616745 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:39.943628 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:39.748616 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" podUID="f4343775-794f-4a3c-8732-01eeff138bbe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:39.981575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:39.971124 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:35:39.984389 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:39.984355 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:40.038991 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:40.030925 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": context deadline exceeded" Oct 30 18:35:40.146572 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:40.107333 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:40.167546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:40.165402 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:40.167546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:40.165482 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" podUID="47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:40.242514 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:40.242404 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:35:40.243472 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:40.243437 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:35:40.252778 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:40.252745 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:35:40.290629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:40.273824 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": context deadline exceeded" Oct 30 18:35:40.339530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:40.338548 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:40.673534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:40.671736 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:35:40.749763 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:40.749725 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:40.750099 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:40.750067 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" podUID="db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:40.789851 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:40.789806 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/health\": context deadline exceeded" Oct 30 18:35:40.909649 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:40.909604 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:35:40.932678 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:40.932578 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:35:40.933560 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:40.933521 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:40.975755 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:40.974602 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.044905 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:41.044081 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.147655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:41.147603 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.252093 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:41.251060 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.252093 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:41.251458 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.254086 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:41.254053 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:35:41.281852 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:41.280609 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.573531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:41.572599 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:41.573531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:41.572662 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" podUID="c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.674640 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:41.674592 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.927163 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:41.927042 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.937648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:41.937582 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:35:41.939401 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:41.939357 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:35:41.978275 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:41.978234 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:42.046474 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:42.046419 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:42.153058 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:42.153014 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": context deadline exceeded" Oct 30 18:35:42.261648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:42.261543 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:42.336530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:42.265169 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:42.336530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:42.330313 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" podUID="d5567046-aa24-421a-8339-fdd9cf56e62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:42.336530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:42.265732 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:42.336530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:42.265719 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:42.336530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:42.311956 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:42.606530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:42.603128 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:42.681562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:42.681518 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:42.937286 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:42.937187 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:35:42.938252 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:42.938218 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:42.960993 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:42.941604 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:42.992009 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:42.991917 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:43.050543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:43.048634 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:43.158542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:43.153604 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:43.333678 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:43.333609 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:43.334383 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:43.334350 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:35:43.334905 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:43.334874 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:35:43.335324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:43.335299 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:35:43.368671 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:43.368631 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:43.368985 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:43.368950 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" podUID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:43.685627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:43.685491 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:43.823894 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:43.823863 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:43.945602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:43.945307 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:44.032603 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:43.985604 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:44.032603 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:43.986005 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:35:44.032603 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.007588 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:35:44.073657 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.068876 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:44.129077 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.128623 4603 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:44.129077 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.128689 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" podUID="22e0b280-2b65-4fad-8344-67049190d108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:44.162770 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.160626 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": context deadline exceeded" Oct 30 18:35:44.188534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.163571 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:44.347528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.320716 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv"] Oct 30 18:35:44.347528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.328305 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.347528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.346904 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:44.347528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.347287 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": context deadline exceeded" Oct 30 18:35:44.347912 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.347604 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:35:44.347912 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.347865 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:35:44.464457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.463903 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-293234-openshift-default-dockercfg-qlf2m\"" Oct 30 18:35:44.479514 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.479024 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:44.485609 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.481591 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:44.485609 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.481653 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" podUID="5411300d-c1aa-4397-be3d-0ce869a1c1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:44.643584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.631029 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Oct 30 18:35:44.805527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.713825 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-workload-socket\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.805527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.713881 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-data\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.805527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.713931 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-token\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.805527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.713961 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqvf7\" (UniqueName: \"kubernetes.io/projected/b6ce7227-01e3-486c-93eb-f945903f32cf-kube-api-access-vqvf7\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.805527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.714052 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-workload-certs\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.805527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.714188 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-podinfo\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.805527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.714319 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6ce7227-01e3-486c-93eb-f945903f32cf-istiod-ca-cert\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.805527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.714415 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-envoy\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.805527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.714544 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-credential-socket\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.805527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.729346 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:44.824615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.820760 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-workload-socket\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.824615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.820814 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-data\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.824615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.820868 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-token\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.824615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.820898 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqvf7\" (UniqueName: \"kubernetes.io/projected/b6ce7227-01e3-486c-93eb-f945903f32cf-kube-api-access-vqvf7\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.824615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.820950 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-workload-certs\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.824615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.821037 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-podinfo\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.824615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.821143 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6ce7227-01e3-486c-93eb-f945903f32cf-istiod-ca-cert\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.824615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.821215 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-envoy\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.824615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.821248 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-credential-socket\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.824615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.821914 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-credential-socket\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.824615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.822653 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-workload-socket\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.824615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.822894 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-data\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.928591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.836579 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6ce7227-01e3-486c-93eb-f945903f32cf-istiod-ca-cert\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.928591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.871990 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-workload-certs\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:44.928591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.895623 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:44.928591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.895683 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" podUID="81f49829-73ed-4760-9444-f1793861eba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:44.928591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.895978 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv"] Oct 30 18:35:44.945651 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.945604 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:44.946901 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.946872 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:44.959805 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:44.959777 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:45.007635 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:45.007587 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:35:45.034791 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:45.034754 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-token\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:45.035006 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:45.024414 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:45.135542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:45.060748 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:45.135542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:45.060818 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" podUID="571fc9dd-71e3-4fa7-b1af-e74f0a96654d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:45.135542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:45.076721 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": context deadline exceeded" Oct 30 18:35:45.216442 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:45.216327 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": context deadline exceeded" Oct 30 18:35:45.236126 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:45.236072 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:45.276213 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:45.268271 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-envoy\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:45.276213 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:45.268850 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-podinfo\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:45.374906 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:45.364754 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:45.374906 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:45.365066 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:35:45.374906 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:45.365254 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": context deadline exceeded" Oct 30 18:35:45.374906 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:45.365419 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:45.522624 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:45.509325 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:45.527534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:45.523519 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqvf7\" (UniqueName: \"kubernetes.io/projected/b6ce7227-01e3-486c-93eb-f945903f32cf-kube-api-access-vqvf7\") pod \"e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:45.762789 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:45.762750 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:45.765402 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:45.765363 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:46.007648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:45.970109 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:46.058166 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:46.048156 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:46.058166 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:46.049038 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:46.090532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:46.078049 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:46.231529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:46.227687 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:46.253541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:46.244628 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:46.443989 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:46.443928 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:46.486546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:46.467472 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:46.486546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:46.467929 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:35:46.486546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:46.468214 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:46.486546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:46.468452 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:46.488055 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:46.488020 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:46.778586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:46.763604 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:47.008891 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:47.008844 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:35:47.087063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:47.057613 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:47.087063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:47.060146 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:47.108806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:47.107136 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": context deadline exceeded" Oct 30 18:35:47.147583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:47.138087 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Oct 30 18:35:47.299533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:47.297592 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:35:47.354566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:47.354448 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:47.474812 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:47.467893 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:47.476538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:47.475986 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:47.477931 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:47.476741 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:35:47.479029 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:47.478099 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:35:47.608805 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:47.602330 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": context deadline exceeded" Oct 30 18:35:47.766651 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:47.766604 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:47.860376 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:47.860273 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:35:47.884529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:47.860741 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:35:47.884965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:47.884925 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:35:48.242486 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.171256 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:48.242486 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.203830 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": context deadline exceeded" Oct 30 18:35:48.242486 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.205677 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:48.458541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.290925 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:35:48.458541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.342690 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:48.458541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.349614 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:35:48.458541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.442789 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:48.485350 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.482749 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:48.485350 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.482848 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:48.485350 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.483918 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:35:48.485738 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.485711 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": context deadline exceeded" Oct 30 18:35:48.525658 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.489051 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:35:48.558249 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.558206 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qhvxv" podUID="f4343775-794f-4a3c-8732-01eeff138bbe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:48.570641 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.570602 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zz7vx" podUID="1112c14a-18f4-46de-ba0c-8e28f19ef80d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:48.656622 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.656577 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:48.668416 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.668380 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-7mmmn" podUID="f0702ace-9c54-4015-add0-e25ee9feaf40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded" Oct 30 18:35:48.670856 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:35:48.670823 4603 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.237s" Oct 30 18:35:48.671016 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.670997 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" event={"ID":"b6ce7227-01e3-486c-93eb-f945903f32cf","Type":"ContainerStarted","Data":"a05705011d885672bd83c602b4d0a856ccf92092774362e3cec45420365f1843"} Oct 30 18:35:48.671140 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.671128 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv"] Oct 30 18:35:48.691036 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.677628 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:48.859684 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.859604 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:35:48.885951 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.885927 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:35:48.886845 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.886824 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:35:48.934710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.930909 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7322-0bf4a7"] Oct 30 18:35:48.934710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.930989 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:35:48.934710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.931939 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7322-0bf4a7" podUID="f0c2b609-d1ed-4ca7-9a35-e63146546568" containerName="test-curl-7322-0bf4a7" containerID="cri-o://cad78504e0695ed56c4aa5bf5e23193a4271ef345bde80325189373a51002709" gracePeriod=2 Oct 30 18:35:48.937193 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.937175 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:35:48.944123 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.944104 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:35:48.993051 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:48.993020 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7322-0bf4a7"] Oct 30 18:35:49.032762 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:49.032735 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:35:49.052286 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:49.037488 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:35:49.052286 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:49.038439 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:35:49.052286 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:49.046098 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:35:49.078847 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:49.078819 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:35:49.534925 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:35:49.534887 4603 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-0bf4a7\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="f0c2b609-d1ed-4ca7-9a35-e63146546568" pod="opendatahub/test-curl-7322-0bf4a7" Oct 30 18:35:49.929557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:49.929461 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:50.036325 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:50.036286 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" event={"ID":"b6ce7227-01e3-486c-93eb-f945903f32cf","Type":"ContainerStarted","Data":"13f370e1c91d6b9858fa11610abd2d2b7a6e100840f20bcefa2e44363dac64be"} Oct 30 18:35:50.102307 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:50.102246 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" podStartSLOduration=6.102227993 podStartE2EDuration="6.102227993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:50.100646256 +0000 UTC m=+2610.816216141" watchObservedRunningTime="2026-10-30 18:35:50.102227993 +0000 UTC m=+2610.817797867" Oct 30 18:35:50.766348 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:50.766316 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:50.772342 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:50.772311 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:51.045132 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:51.045082 4603 generic.go:356] "Generic (PLEG): container finished" podID="f0c2b609-d1ed-4ca7-9a35-e63146546568" containerID="cad78504e0695ed56c4aa5bf5e23193a4271ef345bde80325189373a51002709" exitCode=137 Oct 30 18:35:51.045577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:51.045555 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:51.046830 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:51.046812 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:35:51.269091 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:51.269066 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-0bf4a7" Oct 30 18:35:51.282216 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:35:51.282148 4603 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-0bf4a7\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="f0c2b609-d1ed-4ca7-9a35-e63146546568" pod="opendatahub/test-curl-7322-0bf4a7" Oct 30 18:35:51.394240 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:51.394211 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0c2b609-d1ed-4ca7-9a35-e63146546568-kube-api-access-qbflf\" (UniqueName: \"kubernetes.io/projected/f0c2b609-d1ed-4ca7-9a35-e63146546568-kube-api-access-qbflf\") pod \"f0c2b609-d1ed-4ca7-9a35-e63146546568\" (UID: \"f0c2b609-d1ed-4ca7-9a35-e63146546568\") " Oct 30 18:35:51.396427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:51.396399 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c2b609-d1ed-4ca7-9a35-e63146546568-kube-api-access-qbflf" pod "f0c2b609-d1ed-4ca7-9a35-e63146546568" (UID: "f0c2b609-d1ed-4ca7-9a35-e63146546568"). InnerVolumeSpecName "kube-api-access-qbflf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:35:51.414484 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:51.414453 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0c2b609-d1ed-4ca7-9a35-e63146546568" path="/var/lib/kubelet/pods/f0c2b609-d1ed-4ca7-9a35-e63146546568/volumes" Oct 30 18:35:51.496253 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:51.496221 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbflf\" (UniqueName: \"kubernetes.io/projected/f0c2b609-d1ed-4ca7-9a35-e63146546568-kube-api-access-qbflf\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:35:51.699615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:51.694855 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:51.931492 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:35:51.931454 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:35:52.062323 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:52.062295 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-0bf4a7" Oct 30 18:35:52.062583 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:52.062563 4603 scope.go:122] "RemoveContainer" containerID="cad78504e0695ed56c4aa5bf5e23193a4271ef345bde80325189373a51002709" Oct 30 18:35:52.076139 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:35:52.076114 4603 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-0bf4a7\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="f0c2b609-d1ed-4ca7-9a35-e63146546568" pod="opendatahub/test-curl-7322-0bf4a7" Oct 30 18:35:52.089309 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:35:52.089281 4603 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-0bf4a7\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="f0c2b609-d1ed-4ca7-9a35-e63146546568" pod="opendatahub/test-curl-7322-0bf4a7" Oct 30 18:35:53.329905 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:53.329869 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:55.044839 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:55.044780 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:56.172561 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:56.172496 4603 generic.go:356] "Generic (PLEG): container finished" podID="ec1892e5-3097-4ce7-8138-caceb31225b8" containerID="90c73c94b47ccb54723aa8dbb9d8b36ad0c10d75451978cd5531774413ab2854" exitCode=0 Oct 30 18:35:56.173034 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:56.172572 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" event={"ID":"ec1892e5-3097-4ce7-8138-caceb31225b8","Type":"ContainerDied","Data":"90c73c94b47ccb54723aa8dbb9d8b36ad0c10d75451978cd5531774413ab2854"} Oct 30 18:35:56.173034 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:56.172984 4603 scope.go:122] "RemoveContainer" containerID="90c73c94b47ccb54723aa8dbb9d8b36ad0c10d75451978cd5531774413ab2854" Oct 30 18:35:56.174590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:56.174563 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:35:56.174660 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:56.174630 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:35:56.174700 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:56.174659 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:35:56.644973 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:56.644917 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:57.184686 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.184648 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" event={"ID":"ec1892e5-3097-4ce7-8138-caceb31225b8","Type":"ContainerStarted","Data":"da3fe7c4c0cc06e99c403590fed6447b1ca1780ef1e31d66f0a3ae549525f2f1"} Oct 30 18:35:57.304060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.289285 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6"] Oct 30 18:35:57.305677 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.305642 4603 cpu_manager.go:414] "containerMap: removing container" podUID="f0c2b609-d1ed-4ca7-9a35-e63146546568" containerName="test-curl-7322-0bf4a7" Oct 30 18:35:57.305857 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.305840 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0c2b609-d1ed-4ca7-9a35-e63146546568" containerName="test-curl-7322-0bf4a7" Oct 30 18:35:57.306134 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.306117 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0c2b609-d1ed-4ca7-9a35-e63146546568" containerName="test-curl-7322-0bf4a7" Oct 30 18:35:57.317758 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.317731 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.333465 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.333441 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6"] Oct 30 18:35:57.348899 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.348867 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6p6w\" (UniqueName: \"kubernetes.io/projected/0a8b0573-4657-4fc6-b7fd-43d8e882062d-kube-api-access-r6p6w\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.349214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.348940 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.349313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.349210 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.349386 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.349322 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.349454 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.349392 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.349454 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.349422 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.349600 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.349471 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.349600 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.349497 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.349600 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.349578 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.451203 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.451127 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.451351 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.451218 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.451351 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.451258 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.451351 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.451278 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.451490 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.451347 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.451490 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.451381 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.451490 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.451417 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.451490 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.451477 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6p6w\" (UniqueName: \"kubernetes.io/projected/0a8b0573-4657-4fc6-b7fd-43d8e882062d-kube-api-access-r6p6w\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.451754 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.451534 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.451802 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.451773 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.451841 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.451801 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.452102 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.452080 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.452221 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.452204 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.452368 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.452347 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.453813 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.453795 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.453916 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.453845 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.467924 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.467904 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:35:57.469335 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.469313 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 18:35:57.469434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.469367 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" podUID="ec1892e5-3097-4ce7-8138-caceb31225b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 18:35:57.473941 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.473922 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.474196 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.474175 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6p6w\" (UniqueName: \"kubernetes.io/projected/0a8b0573-4657-4fc6-b7fd-43d8e882062d-kube-api-access-r6p6w\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.635678 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.635647 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:57.980087 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.980057 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6"] Oct 30 18:35:57.980432 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:35:57.980405 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8b0573_4657_4fc6_b7fd_43d8e882062d.slice/crio-48c01fcbd28167ba71e40c89aa7a7585b6100333030532d2c02b9cfdf63feb7d WatchSource:0}: Error finding container 48c01fcbd28167ba71e40c89aa7a7585b6100333030532d2c02b9cfdf63feb7d: Status 404 returned error can't find the container with id 48c01fcbd28167ba71e40c89aa7a7585b6100333030532d2c02b9cfdf63feb7d Oct 30 18:35:57.983255 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.983221 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:35:57.983330 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.983291 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:35:57.983330 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:57.983322 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:35:58.211893 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:58.211772 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" event={"ID":"0a8b0573-4657-4fc6-b7fd-43d8e882062d","Type":"ContainerStarted","Data":"d9e69bc11a19f762f294b24444ef876756df3428736b377eefde88c1f7b152bf"} Oct 30 18:35:58.211893 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:58.211850 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" event={"ID":"0a8b0573-4657-4fc6-b7fd-43d8e882062d","Type":"ContainerStarted","Data":"48c01fcbd28167ba71e40c89aa7a7585b6100333030532d2c02b9cfdf63feb7d"} Oct 30 18:35:58.259765 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:58.259663 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" podStartSLOduration=1.259647457 podStartE2EDuration="1.259647457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:58.257226183 +0000 UTC m=+2618.972796057" watchObservedRunningTime="2026-10-30 18:35:58.259647457 +0000 UTC m=+2618.975217331" Oct 30 18:35:58.328305 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:58.328226 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:58.473529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:58.470074 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 18:35:58.473529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:58.470138 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" podUID="ec1892e5-3097-4ce7-8138-caceb31225b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 18:35:58.635995 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:58.635963 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:35:59.467585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:59.467555 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 18:35:59.468048 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:59.468024 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" podUID="ec1892e5-3097-4ce7-8138-caceb31225b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 18:35:59.641525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:59.638913 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 18:35:59.641525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:59.638972 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" podUID="0a8b0573-4657-4fc6-b7fd-43d8e882062d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 18:35:59.679295 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:59.679263 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:59.876529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:35:59.872827 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:00.143613 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:00.141027 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:00.521588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:00.520829 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:36:00.529765 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:00.529729 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 18:36:00.530057 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:00.530017 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" podUID="ec1892e5-3097-4ce7-8138-caceb31225b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 18:36:00.643523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:00.637256 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 18:36:00.643523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:00.637330 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" podUID="0a8b0573-4657-4fc6-b7fd-43d8e882062d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 18:36:00.713424 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:00.706674 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:01.049624 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.045122 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt"] Oct 30 18:36:01.054286 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.054263 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt"] Oct 30 18:36:01.054567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.054553 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.111530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.088109 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-c8eef3-openshift-default-dockercfg-4hdd4\"" Oct 30 18:36:01.156522 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.155492 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-workload-certs\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.156522 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.155567 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-credential-socket\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.156522 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.155679 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-token\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.156522 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.155728 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-podinfo\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.156522 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.155807 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bcc5bf-0143-429f-95c1-298df70d5f88-istiod-ca-cert\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.156522 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.155969 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-envoy\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.156522 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.155999 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-workload-socket\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.156522 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.156041 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-data\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.156522 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.156132 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c88h\" (UniqueName: \"kubernetes.io/projected/f9bcc5bf-0143-429f-95c1-298df70d5f88-kube-api-access-6c88h\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.159577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.159549 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46"] Oct 30 18:36:01.180294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.178299 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:01.279528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.277771 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-token\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.279528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.277847 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-podinfo\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.279528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.277960 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bcc5bf-0143-429f-95c1-298df70d5f88-istiod-ca-cert\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.279528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.278216 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-envoy\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.279528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.278259 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-workload-socket\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.279528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.278317 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-data\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.279528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.278435 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c88h\" (UniqueName: \"kubernetes.io/projected/f9bcc5bf-0143-429f-95c1-298df70d5f88-kube-api-access-6c88h\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.284598 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.284348 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bcc5bf-0143-429f-95c1-298df70d5f88-istiod-ca-cert\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.296527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.288917 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-workload-socket\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.296527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.289828 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-data\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.326934 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.324193 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-podinfo\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.346391 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.340393 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c88h\" (UniqueName: \"kubernetes.io/projected/f9bcc5bf-0143-429f-95c1-298df70d5f88-kube-api-access-6c88h\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.346391 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.340625 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-token\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.346391 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.344670 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-workload-certs\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.346391 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.344727 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-credential-socket\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.346391 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.345316 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-credential-socket\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.346391 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.345596 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-workload-certs\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.380599 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.350629 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-envoy\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.390866 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.390833 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:01.445568 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.441715 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:01.470994 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.470949 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 18:36:01.471288 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.471262 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" podUID="ec1892e5-3097-4ce7-8138-caceb31225b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 18:36:01.653272 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.653236 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 18:36:01.653887 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.653856 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" podUID="0a8b0573-4657-4fc6-b7fd-43d8e882062d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 18:36:01.655688 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:01.655663 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:02.145747 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:36:02.145703 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9bcc5bf_0143_429f_95c1_298df70d5f88.slice/crio-5cb9d770b3ba62381673fc0417d3e562b4b79209aa25ef6b3e024272415006ae WatchSource:0}: Error finding container 5cb9d770b3ba62381673fc0417d3e562b4b79209aa25ef6b3e024272415006ae: Status 404 returned error can't find the container with id 5cb9d770b3ba62381673fc0417d3e562b4b79209aa25ef6b3e024272415006ae Oct 30 18:36:02.146360 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:02.146336 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt"] Oct 30 18:36:02.148975 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:02.148946 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:36:02.149051 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:02.149031 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:36:02.149093 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:02.149061 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:36:02.467978 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:02.467903 4603 patch_prober.go:29] interesting pod/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 18:36:02.467978 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:02.467957 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" podUID="ec1892e5-3097-4ce7-8138-caceb31225b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 18:36:02.643529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:02.639882 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 18:36:02.643529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:02.639939 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" podUID="0a8b0573-4657-4fc6-b7fd-43d8e882062d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 18:36:02.665128 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:02.665083 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" event={"ID":"f9bcc5bf-0143-429f-95c1-298df70d5f88","Type":"ContainerStarted","Data":"dd9711d5b66007469ce86d91263f92ac85c307a18897fff91c3361931f121022"} Oct 30 18:36:02.665585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:02.665186 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" event={"ID":"f9bcc5bf-0143-429f-95c1-298df70d5f88","Type":"ContainerStarted","Data":"5cb9d770b3ba62381673fc0417d3e562b4b79209aa25ef6b3e024272415006ae"} Oct 30 18:36:02.912000 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:02.911961 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:03.079699 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:03.079666 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:03.407529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:03.397143 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:03.407529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:03.397323 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 18:36:03.407529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:03.397377 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" podUID="f9bcc5bf-0143-429f-95c1-298df70d5f88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 18:36:03.636927 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:03.636889 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 18:36:03.637213 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:03.637184 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" podUID="0a8b0573-4657-4fc6-b7fd-43d8e882062d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 18:36:03.659086 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:03.659006 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:36:03.660791 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:03.660767 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:36:03.744531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:03.744423 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" podStartSLOduration=3.744402833 podStartE2EDuration="3.744402833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:02.714820667 +0000 UTC m=+2623.430390576" watchObservedRunningTime="2026-10-30 18:36:03.744402833 +0000 UTC m=+2624.459972710" Oct 30 18:36:04.438280 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:04.438224 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 18:36:04.438493 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:04.438453 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" podUID="f9bcc5bf-0143-429f-95c1-298df70d5f88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 18:36:04.649558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:04.647090 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 18:36:04.649558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:04.647148 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" podUID="0a8b0573-4657-4fc6-b7fd-43d8e882062d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 18:36:04.706442 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:04.706353 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:04.861897 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:04.861833 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:05.450546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:05.431082 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 18:36:05.450546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:05.449985 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" podUID="f9bcc5bf-0143-429f-95c1-298df70d5f88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 18:36:05.636309 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:05.636274 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 18:36:05.636492 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:05.636341 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" podUID="0a8b0573-4657-4fc6-b7fd-43d8e882062d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 18:36:06.399885 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.399852 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 18:36:06.400483 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.400454 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" podUID="f9bcc5bf-0143-429f-95c1-298df70d5f88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 18:36:06.424461 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.424421 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4"] Oct 30 18:36:06.431940 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.431908 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.447566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.447536 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4"] Oct 30 18:36:06.543906 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.543869 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:06.613536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.573910 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92796f5-3580-47ed-bcde-da5fb7075252-istio-podinfo\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.614004 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.613975 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-istio-envoy\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.614394 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.614366 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-workload-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.614690 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.614668 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-credential-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.614863 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.614847 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-istio-data\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.616437 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.616410 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92796f5-3580-47ed-bcde-da5fb7075252-istiod-ca-cert\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.627783 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.627749 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f88g2\" (UniqueName: \"kubernetes.io/projected/e92796f5-3580-47ed-bcde-da5fb7075252-kube-api-access-f88g2\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.628119 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.628092 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-workload-certs\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.652581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.628291 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92796f5-3580-47ed-bcde-da5fb7075252-istio-token\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.661529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.658340 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:36:06.661529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.660545 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:36:06.674532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.669773 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:36:06.757859 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.757781 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-istio-data\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.758352 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.758322 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92796f5-3580-47ed-bcde-da5fb7075252-istiod-ca-cert\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.758563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.758537 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-istio-data\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.758715 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.758694 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f88g2\" (UniqueName: \"kubernetes.io/projected/e92796f5-3580-47ed-bcde-da5fb7075252-kube-api-access-f88g2\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.758925 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.758899 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-workload-certs\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.759017 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.758947 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92796f5-3580-47ed-bcde-da5fb7075252-istio-token\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.759080 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.759019 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92796f5-3580-47ed-bcde-da5fb7075252-istio-podinfo\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.759212 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.759195 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-istio-envoy\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.759417 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.759401 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-workload-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.759565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.759549 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-credential-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.759910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.759889 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-credential-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.760393 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.760372 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92796f5-3580-47ed-bcde-da5fb7075252-istiod-ca-cert\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.785948 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.781121 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-workload-certs\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.785948 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.781899 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-workload-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.794965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.794927 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92796f5-3580-47ed-bcde-da5fb7075252-istio-podinfo\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.831751 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.831713 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-istio-envoy\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.847539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.835288 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f88g2\" (UniqueName: \"kubernetes.io/projected/e92796f5-3580-47ed-bcde-da5fb7075252-kube-api-access-f88g2\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:06.847539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:06.836328 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92796f5-3580-47ed-bcde-da5fb7075252-istio-token\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:07.051422 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:07.051387 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:07.416703 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:07.416557 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 18:36:07.416703 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:07.416617 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" podUID="f9bcc5bf-0143-429f-95c1-298df70d5f88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 18:36:07.420442 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:36:07.420404 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92796f5_3580_47ed_bcde_da5fb7075252.slice/crio-2b9f3bcac4336763c47254b83ce901ca220c5322bbdad7a27984082bc94f0be2 WatchSource:0}: Error finding container 2b9f3bcac4336763c47254b83ce901ca220c5322bbdad7a27984082bc94f0be2: Status 404 returned error can't find the container with id 2b9f3bcac4336763c47254b83ce901ca220c5322bbdad7a27984082bc94f0be2 Oct 30 18:36:07.423286 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:07.423255 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:36:07.423375 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:07.423324 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:36:07.423375 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:07.423353 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:36:07.423701 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:07.423681 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4"] Oct 30 18:36:07.760977 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:07.760881 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" event={"ID":"e92796f5-3580-47ed-bcde-da5fb7075252","Type":"ContainerStarted","Data":"3b374f6038729b8f36db585561f52e7a6e2081fa6224d206960b00ac8934c2af"} Oct 30 18:36:07.760977 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:07.760932 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" event={"ID":"e92796f5-3580-47ed-bcde-da5fb7075252","Type":"ContainerStarted","Data":"2b9f3bcac4336763c47254b83ce901ca220c5322bbdad7a27984082bc94f0be2"} Oct 30 18:36:07.816529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:07.815601 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" podStartSLOduration=1.815566915 podStartE2EDuration="1.815566915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:07.804940114 +0000 UTC m=+2628.520510017" watchObservedRunningTime="2026-10-30 18:36:07.815566915 +0000 UTC m=+2628.531136789" Oct 30 18:36:08.051682 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:08.051649 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:08.052756 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:08.052728 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:36:08.052903 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:08.052783 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" podUID="e92796f5-3580-47ed-bcde-da5fb7075252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:36:08.238624 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:08.238587 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:08.402581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:08.397852 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 18:36:08.402581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:08.397924 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" podUID="f9bcc5bf-0143-429f-95c1-298df70d5f88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 18:36:09.052615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:09.052579 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:36:09.053014 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:09.052640 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" podUID="e92796f5-3580-47ed-bcde-da5fb7075252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:36:09.414584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:09.408927 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 18:36:09.414584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:09.408990 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" podUID="f9bcc5bf-0143-429f-95c1-298df70d5f88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 18:36:09.951525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:09.950591 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:10.054090 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:10.054028 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:36:10.054754 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:10.054708 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" podUID="e92796f5-3580-47ed-bcde-da5fb7075252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:36:10.418858 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:10.418826 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:10.419212 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:10.419191 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:10.419857 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:10.419837 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:36:11.052627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:11.052592 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:36:11.052822 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:11.052667 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" podUID="e92796f5-3580-47ed-bcde-da5fb7075252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:36:11.730939 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:11.730892 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:12.052744 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:12.052708 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:36:12.053006 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:12.052771 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" podUID="e92796f5-3580-47ed-bcde-da5fb7075252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:36:12.115240 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:12.115199 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:13.052545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:13.052497 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:36:13.053039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:13.052562 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" podUID="e92796f5-3580-47ed-bcde-da5fb7075252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:36:13.505579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:13.499987 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:14.052093 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:14.052048 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:36:14.052303 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:14.052149 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" podUID="e92796f5-3580-47ed-bcde-da5fb7075252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:36:15.057726 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:15.057701 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:15.058213 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:15.058127 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:15.058914 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:15.058894 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:36:15.183693 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:15.183654 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:16.188059 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.188018 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" containerID="cri-o://2c1c1f39a54cb8aebcf2dbf7f6f64c5d2975cfec95267ab955ec321eb6f6cddd" gracePeriod=45 Oct 30 18:36:16.469003 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.468971 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq"] Oct 30 18:36:16.484628 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.484600 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:36:16.541027 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.540985 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:16.582555 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.575216 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a35d239-64fb-4752-91e9-10f61d87713f-kube-api-access-wjtpf\" (UniqueName: \"kubernetes.io/projected/2a35d239-64fb-4752-91e9-10f61d87713f-kube-api-access-wjtpf\") pod \"2a35d239-64fb-4752-91e9-10f61d87713f\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " Oct 30 18:36:16.582555 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.575456 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-model-cache\") pod \"2a35d239-64fb-4752-91e9-10f61d87713f\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " Oct 30 18:36:16.582555 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.575490 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a35d239-64fb-4752-91e9-10f61d87713f-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a35d239-64fb-4752-91e9-10f61d87713f-tls-certs\") pod \"2a35d239-64fb-4752-91e9-10f61d87713f\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " Oct 30 18:36:16.582555 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.575550 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-home\" (UniqueName: \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-home\") pod \"2a35d239-64fb-4752-91e9-10f61d87713f\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " Oct 30 18:36:16.582555 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.575631 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-dshm\") pod \"2a35d239-64fb-4752-91e9-10f61d87713f\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " Oct 30 18:36:16.582555 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.575725 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-tmp-dir\") pod \"2a35d239-64fb-4752-91e9-10f61d87713f\" (UID: \"2a35d239-64fb-4752-91e9-10f61d87713f\") " Oct 30 18:36:16.608201 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.605641 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-home" pod "2a35d239-64fb-4752-91e9-10f61d87713f" (UID: "2a35d239-64fb-4752-91e9-10f61d87713f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:16.608201 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.605935 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-model-cache" pod "2a35d239-64fb-4752-91e9-10f61d87713f" (UID: "2a35d239-64fb-4752-91e9-10f61d87713f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:16.608201 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.606548 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-tmp-dir" pod "2a35d239-64fb-4752-91e9-10f61d87713f" (UID: "2a35d239-64fb-4752-91e9-10f61d87713f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:16.703580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.701296 4603 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-model-cache\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:36:16.703580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.701341 4603 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-home\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:36:16.703580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.701360 4603 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-tmp-dir\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:36:16.777547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.776743 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a35d239-64fb-4752-91e9-10f61d87713f-kube-api-access-wjtpf" pod "2a35d239-64fb-4752-91e9-10f61d87713f" (UID: "2a35d239-64fb-4752-91e9-10f61d87713f"). InnerVolumeSpecName "kube-api-access-wjtpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:16.815192 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.808325 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjtpf\" (UniqueName: \"kubernetes.io/projected/2a35d239-64fb-4752-91e9-10f61d87713f-kube-api-access-wjtpf\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:36:16.828910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.828848 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-dshm" pod "2a35d239-64fb-4752-91e9-10f61d87713f" (UID: "2a35d239-64fb-4752-91e9-10f61d87713f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:16.829299 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.829280 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a35d239-64fb-4752-91e9-10f61d87713f-tls-certs" pod "2a35d239-64fb-4752-91e9-10f61d87713f" (UID: "2a35d239-64fb-4752-91e9-10f61d87713f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:16.877444 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.877414 4603 generic.go:356] "Generic (PLEG): container finished" podID="2a35d239-64fb-4752-91e9-10f61d87713f" containerID="2c1c1f39a54cb8aebcf2dbf7f6f64c5d2975cfec95267ab955ec321eb6f6cddd" exitCode=0 Oct 30 18:36:16.877648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.877459 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" Oct 30 18:36:16.877648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.877524 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" event={"ID":"2a35d239-64fb-4752-91e9-10f61d87713f","Type":"ContainerDied","Data":"2c1c1f39a54cb8aebcf2dbf7f6f64c5d2975cfec95267ab955ec321eb6f6cddd"} Oct 30 18:36:16.877648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.877575 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46" event={"ID":"2a35d239-64fb-4752-91e9-10f61d87713f","Type":"ContainerDied","Data":"e41e427be3eded20f438b2e64cb83576e5f5914094ee1a3be2ff95f468e54b17"} Oct 30 18:36:16.877648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.877596 4603 scope.go:122] "RemoveContainer" containerID="2c1c1f39a54cb8aebcf2dbf7f6f64c5d2975cfec95267ab955ec321eb6f6cddd" Oct 30 18:36:16.888194 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.888175 4603 scope.go:122] "RemoveContainer" containerID="2c1c1f39a54cb8aebcf2dbf7f6f64c5d2975cfec95267ab955ec321eb6f6cddd" Oct 30 18:36:16.888448 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:36:16.888429 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c1c1f39a54cb8aebcf2dbf7f6f64c5d2975cfec95267ab955ec321eb6f6cddd\": container with ID starting with 2c1c1f39a54cb8aebcf2dbf7f6f64c5d2975cfec95267ab955ec321eb6f6cddd not found: ID does not exist" containerID="2c1c1f39a54cb8aebcf2dbf7f6f64c5d2975cfec95267ab955ec321eb6f6cddd" Oct 30 18:36:16.888536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.888462 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c1c1f39a54cb8aebcf2dbf7f6f64c5d2975cfec95267ab955ec321eb6f6cddd"} err="failed to get container status \"2c1c1f39a54cb8aebcf2dbf7f6f64c5d2975cfec95267ab955ec321eb6f6cddd\": rpc error: code = NotFound desc = could not find container \"2c1c1f39a54cb8aebcf2dbf7f6f64c5d2975cfec95267ab955ec321eb6f6cddd\": container with ID starting with 2c1c1f39a54cb8aebcf2dbf7f6f64c5d2975cfec95267ab955ec321eb6f6cddd not found: ID does not exist" Oct 30 18:36:16.910364 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.910344 4603 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a35d239-64fb-4752-91e9-10f61d87713f-tls-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:36:16.910364 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.910365 4603 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a35d239-64fb-4752-91e9-10f61d87713f-dshm\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:36:16.930311 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.930289 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46"] Oct 30 18:36:16.935637 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:16.935616 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a2a10a/auth-test-model-a2a10a-kserve-867f45cf6f-sqh46"] Oct 30 18:36:17.068138 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:17.068107 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:17.413533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:17.413436 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" path="/var/lib/kubelet/pods/2a35d239-64fb-4752-91e9-10f61d87713f/volumes" Oct 30 18:36:18.123883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:18.123850 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:18.750821 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:18.741238 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:18.899648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:18.899617 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:18.984227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:18.984187 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v"] Oct 30 18:36:18.984463 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:18.984421 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" podUID="67142e99-ffa0-4b21-8f93-735d5663c87e" containerName="istio-proxy" containerID="cri-o://c2e092dbe8ec0eeb04e8b29e3f37c0d574a81c98465b9c1d6e570fc363785e83" gracePeriod=30 Oct 30 18:36:19.044555 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:19.044472 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:19.201546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:19.186912 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:20.619056 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:20.608655 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:21.933900 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:36:21.933866 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:36:22.321179 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:22.321146 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:23.988418 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:23.986962 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:24.300952 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.300926 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:36:24.482475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.482445 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-credential-socket\") pod \"67142e99-ffa0-4b21-8f93-735d5663c87e\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " Oct 30 18:36:24.482671 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.482488 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-podinfo\") pod \"67142e99-ffa0-4b21-8f93-735d5663c87e\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " Oct 30 18:36:24.482671 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.482528 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67142e99-ffa0-4b21-8f93-735d5663c87e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67142e99-ffa0-4b21-8f93-735d5663c87e-istiod-ca-cert\") pod \"67142e99-ffa0-4b21-8f93-735d5663c87e\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " Oct 30 18:36:24.482671 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.482588 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-workload-socket\") pod \"67142e99-ffa0-4b21-8f93-735d5663c87e\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " Oct 30 18:36:24.482671 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.482647 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-workload-certs\") pod \"67142e99-ffa0-4b21-8f93-735d5663c87e\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " Oct 30 18:36:24.482907 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.482728 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-credential-socket" pod "67142e99-ffa0-4b21-8f93-735d5663c87e" (UID: "67142e99-ffa0-4b21-8f93-735d5663c87e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:24.482907 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.482744 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-data\") pod \"67142e99-ffa0-4b21-8f93-735d5663c87e\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " Oct 30 18:36:24.482907 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.482886 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-envoy\") pod \"67142e99-ffa0-4b21-8f93-735d5663c87e\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " Oct 30 18:36:24.483668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.482921 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-workload-certs" pod "67142e99-ffa0-4b21-8f93-735d5663c87e" (UID: "67142e99-ffa0-4b21-8f93-735d5663c87e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:24.483668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.482946 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-workload-socket" pod "67142e99-ffa0-4b21-8f93-735d5663c87e" (UID: "67142e99-ffa0-4b21-8f93-735d5663c87e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:24.483668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.482982 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67142e99-ffa0-4b21-8f93-735d5663c87e-istiod-ca-cert" pod "67142e99-ffa0-4b21-8f93-735d5663c87e" (UID: "67142e99-ffa0-4b21-8f93-735d5663c87e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:24.483668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.482953 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67142e99-ffa0-4b21-8f93-735d5663c87e-kube-api-access-2fh94\" (UniqueName: \"kubernetes.io/projected/67142e99-ffa0-4b21-8f93-735d5663c87e-kube-api-access-2fh94\") pod \"67142e99-ffa0-4b21-8f93-735d5663c87e\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " Oct 30 18:36:24.483668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.483022 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-data" pod "67142e99-ffa0-4b21-8f93-735d5663c87e" (UID: "67142e99-ffa0-4b21-8f93-735d5663c87e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:24.483668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.483113 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-token\" (UniqueName: \"kubernetes.io/projected/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-token\") pod \"67142e99-ffa0-4b21-8f93-735d5663c87e\" (UID: \"67142e99-ffa0-4b21-8f93-735d5663c87e\") " Oct 30 18:36:24.483996 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.483864 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:36:24.483996 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.483888 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67142e99-ffa0-4b21-8f93-735d5663c87e-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:36:24.483996 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.483903 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:36:24.483996 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.483918 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:36:24.483996 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.483932 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:36:24.485145 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.485120 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-envoy" pod "67142e99-ffa0-4b21-8f93-735d5663c87e" (UID: "67142e99-ffa0-4b21-8f93-735d5663c87e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:24.485272 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.485224 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67142e99-ffa0-4b21-8f93-735d5663c87e-kube-api-access-2fh94" pod "67142e99-ffa0-4b21-8f93-735d5663c87e" (UID: "67142e99-ffa0-4b21-8f93-735d5663c87e"). InnerVolumeSpecName "kube-api-access-2fh94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:24.485621 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.485596 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-podinfo" pod "67142e99-ffa0-4b21-8f93-735d5663c87e" (UID: "67142e99-ffa0-4b21-8f93-735d5663c87e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:36:24.485708 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.485601 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-token" pod "67142e99-ffa0-4b21-8f93-735d5663c87e" (UID: "67142e99-ffa0-4b21-8f93-735d5663c87e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:24.586223 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.586202 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:36:24.586223 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.586224 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fh94\" (UniqueName: \"kubernetes.io/projected/67142e99-ffa0-4b21-8f93-735d5663c87e-kube-api-access-2fh94\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:36:24.586408 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.586235 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:36:24.586408 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.586244 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67142e99-ffa0-4b21-8f93-735d5663c87e-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:36:24.940652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.940553 4603 generic.go:356] "Generic (PLEG): container finished" podID="67142e99-ffa0-4b21-8f93-735d5663c87e" containerID="c2e092dbe8ec0eeb04e8b29e3f37c0d574a81c98465b9c1d6e570fc363785e83" exitCode=0 Oct 30 18:36:24.940652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.940635 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" Oct 30 18:36:24.940896 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.940710 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" event={"ID":"67142e99-ffa0-4b21-8f93-735d5663c87e","Type":"ContainerDied","Data":"c2e092dbe8ec0eeb04e8b29e3f37c0d574a81c98465b9c1d6e570fc363785e83"} Oct 30 18:36:24.940896 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.940751 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v" event={"ID":"67142e99-ffa0-4b21-8f93-735d5663c87e","Type":"ContainerDied","Data":"eb8a220d7af35432580047a45b424e24f681a2732104ac748f66cc209070ae5b"} Oct 30 18:36:24.940896 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.940778 4603 scope.go:122] "RemoveContainer" containerID="c2e092dbe8ec0eeb04e8b29e3f37c0d574a81c98465b9c1d6e570fc363785e83" Oct 30 18:36:24.953114 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.953094 4603 scope.go:122] "RemoveContainer" containerID="8aa33ec0ada9b6affb208856da6a1cfbbfa39a7f97414d9d8cc44e1bd2b210f6" Oct 30 18:36:24.962754 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.962738 4603 scope.go:122] "RemoveContainer" containerID="c2e092dbe8ec0eeb04e8b29e3f37c0d574a81c98465b9c1d6e570fc363785e83" Oct 30 18:36:24.963019 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:36:24.963003 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e092dbe8ec0eeb04e8b29e3f37c0d574a81c98465b9c1d6e570fc363785e83\": container with ID starting with c2e092dbe8ec0eeb04e8b29e3f37c0d574a81c98465b9c1d6e570fc363785e83 not found: ID does not exist" containerID="c2e092dbe8ec0eeb04e8b29e3f37c0d574a81c98465b9c1d6e570fc363785e83" Oct 30 18:36:24.963061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.963028 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e092dbe8ec0eeb04e8b29e3f37c0d574a81c98465b9c1d6e570fc363785e83"} err="failed to get container status \"c2e092dbe8ec0eeb04e8b29e3f37c0d574a81c98465b9c1d6e570fc363785e83\": rpc error: code = NotFound desc = could not find container \"c2e092dbe8ec0eeb04e8b29e3f37c0d574a81c98465b9c1d6e570fc363785e83\": container with ID starting with c2e092dbe8ec0eeb04e8b29e3f37c0d574a81c98465b9c1d6e570fc363785e83 not found: ID does not exist" Oct 30 18:36:24.963061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.963045 4603 scope.go:122] "RemoveContainer" containerID="8aa33ec0ada9b6affb208856da6a1cfbbfa39a7f97414d9d8cc44e1bd2b210f6" Oct 30 18:36:24.963278 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:36:24.963263 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aa33ec0ada9b6affb208856da6a1cfbbfa39a7f97414d9d8cc44e1bd2b210f6\": container with ID starting with 8aa33ec0ada9b6affb208856da6a1cfbbfa39a7f97414d9d8cc44e1bd2b210f6 not found: ID does not exist" containerID="8aa33ec0ada9b6affb208856da6a1cfbbfa39a7f97414d9d8cc44e1bd2b210f6" Oct 30 18:36:24.963361 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:24.963283 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa33ec0ada9b6affb208856da6a1cfbbfa39a7f97414d9d8cc44e1bd2b210f6"} err="failed to get container status \"8aa33ec0ada9b6affb208856da6a1cfbbfa39a7f97414d9d8cc44e1bd2b210f6\": rpc error: code = NotFound desc = could not find container \"8aa33ec0ada9b6affb208856da6a1cfbbfa39a7f97414d9d8cc44e1bd2b210f6\": container with ID starting with 8aa33ec0ada9b6affb208856da6a1cfbbfa39a7f97414d9d8cc44e1bd2b210f6 not found: ID does not exist" Oct 30 18:36:25.006155 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:25.006105 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v"] Oct 30 18:36:25.017722 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:25.017699 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-rp85v"] Oct 30 18:36:25.412659 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:25.412616 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67142e99-ffa0-4b21-8f93-735d5663c87e" path="/var/lib/kubelet/pods/67142e99-ffa0-4b21-8f93-735d5663c87e/volumes" Oct 30 18:36:25.650189 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:25.650156 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:27.277868 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:27.277836 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:27.420216 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:27.420183 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:27.678538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:27.678402 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:28.665104 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:28.665061 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:28.843542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:28.843471 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:29.179513 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:29.179470 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:30.988452 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:30.988417 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:31.494159 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:31.494101 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" containerID="cri-o://91951558205df012a4ce56c76f19d13d9a8fe05fcad84b323706e3529a340bb7" gracePeriod=45 Oct 30 18:36:31.789832 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:31.789802 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:36:31.881705 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:31.881675 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc4f5be5-e07f-4960-ace5-4023198470be-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc4f5be5-e07f-4960-ace5-4023198470be-tls-certs\") pod \"cc4f5be5-e07f-4960-ace5-4023198470be\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " Oct 30 18:36:31.881875 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:31.881741 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc4f5be5-e07f-4960-ace5-4023198470be-kube-api-access-lwvtz\" (UniqueName: \"kubernetes.io/projected/cc4f5be5-e07f-4960-ace5-4023198470be-kube-api-access-lwvtz\") pod \"cc4f5be5-e07f-4960-ace5-4023198470be\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " Oct 30 18:36:31.881875 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:31.881846 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-dshm\") pod \"cc4f5be5-e07f-4960-ace5-4023198470be\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " Oct 30 18:36:31.882122 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:31.882101 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-tmp-dir\") pod \"cc4f5be5-e07f-4960-ace5-4023198470be\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " Oct 30 18:36:31.882425 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:31.882395 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-tmp-dir" pod "cc4f5be5-e07f-4960-ace5-4023198470be" (UID: "cc4f5be5-e07f-4960-ace5-4023198470be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:31.882789 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:31.882763 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-model-cache" pod "cc4f5be5-e07f-4960-ace5-4023198470be" (UID: "cc4f5be5-e07f-4960-ace5-4023198470be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:31.882923 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:31.882368 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-model-cache\") pod \"cc4f5be5-e07f-4960-ace5-4023198470be\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " Oct 30 18:36:31.882992 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:31.882964 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-home\" (UniqueName: \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-home\") pod \"cc4f5be5-e07f-4960-ace5-4023198470be\" (UID: \"cc4f5be5-e07f-4960-ace5-4023198470be\") " Oct 30 18:36:31.883226 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:31.883201 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-home" pod "cc4f5be5-e07f-4960-ace5-4023198470be" (UID: "cc4f5be5-e07f-4960-ace5-4023198470be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:31.884084 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:31.884059 4603 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-home\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:36:31.884194 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:31.884088 4603 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-tmp-dir\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:36:31.884194 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:31.884101 4603 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-model-cache\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:36:31.884750 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:31.884718 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc4f5be5-e07f-4960-ace5-4023198470be-tls-certs" pod "cc4f5be5-e07f-4960-ace5-4023198470be" (UID: "cc4f5be5-e07f-4960-ace5-4023198470be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:31.884865 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:31.884841 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc4f5be5-e07f-4960-ace5-4023198470be-kube-api-access-lwvtz" pod "cc4f5be5-e07f-4960-ace5-4023198470be" (UID: "cc4f5be5-e07f-4960-ace5-4023198470be"). InnerVolumeSpecName "kube-api-access-lwvtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:31.884928 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:31.884893 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-dshm" pod "cc4f5be5-e07f-4960-ace5-4023198470be" (UID: "cc4f5be5-e07f-4960-ace5-4023198470be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:31.985666 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:31.985639 4603 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc4f5be5-e07f-4960-ace5-4023198470be-tls-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:36:31.985666 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:31.985666 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwvtz\" (UniqueName: \"kubernetes.io/projected/cc4f5be5-e07f-4960-ace5-4023198470be-kube-api-access-lwvtz\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:36:31.985851 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:31.985676 4603 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc4f5be5-e07f-4960-ace5-4023198470be-dshm\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:36:32.073726 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:32.073691 4603 generic.go:356] "Generic (PLEG): container finished" podID="cc4f5be5-e07f-4960-ace5-4023198470be" containerID="91951558205df012a4ce56c76f19d13d9a8fe05fcad84b323706e3529a340bb7" exitCode=0 Oct 30 18:36:32.074144 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:32.073737 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" Oct 30 18:36:32.074144 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:32.073774 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" event={"ID":"cc4f5be5-e07f-4960-ace5-4023198470be","Type":"ContainerDied","Data":"91951558205df012a4ce56c76f19d13d9a8fe05fcad84b323706e3529a340bb7"} Oct 30 18:36:32.074144 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:32.073823 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq" event={"ID":"cc4f5be5-e07f-4960-ace5-4023198470be","Type":"ContainerDied","Data":"03b7a6b6f3408300e0c4d33bb7af8f7ca961dd94ebe47025645d28ffa9dcbdb4"} Oct 30 18:36:32.074144 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:32.073842 4603 scope.go:122] "RemoveContainer" containerID="91951558205df012a4ce56c76f19d13d9a8fe05fcad84b323706e3529a340bb7" Oct 30 18:36:32.084536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:32.084520 4603 scope.go:122] "RemoveContainer" containerID="91951558205df012a4ce56c76f19d13d9a8fe05fcad84b323706e3529a340bb7" Oct 30 18:36:32.084787 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:36:32.084769 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91951558205df012a4ce56c76f19d13d9a8fe05fcad84b323706e3529a340bb7\": container with ID starting with 91951558205df012a4ce56c76f19d13d9a8fe05fcad84b323706e3529a340bb7 not found: ID does not exist" containerID="91951558205df012a4ce56c76f19d13d9a8fe05fcad84b323706e3529a340bb7" Oct 30 18:36:32.084838 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:32.084795 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91951558205df012a4ce56c76f19d13d9a8fe05fcad84b323706e3529a340bb7"} err="failed to get container status \"91951558205df012a4ce56c76f19d13d9a8fe05fcad84b323706e3529a340bb7\": rpc error: code = NotFound desc = could not find container \"91951558205df012a4ce56c76f19d13d9a8fe05fcad84b323706e3529a340bb7\": container with ID starting with 91951558205df012a4ce56c76f19d13d9a8fe05fcad84b323706e3529a340bb7 not found: ID does not exist" Oct 30 18:36:32.133151 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:32.133129 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq"] Oct 30 18:36:32.143634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:32.143616 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b00d69/auth-test-model-b00d69-kserve-5ff49f965c-8l5zq"] Oct 30 18:36:32.650433 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:32.650386 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:33.410734 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:33.410698 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" path="/var/lib/kubelet/pods/cc4f5be5-e07f-4960-ace5-4023198470be/volumes" Oct 30 18:36:34.299911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:34.297023 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:35.044235 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:35.044202 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:35.191431 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:35.191392 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:36.041282 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:36.035477 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:36.799528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:36.792519 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:37.780549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:37.780496 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:37.922527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:37.922425 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:39.761242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:39.761193 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:40.350912 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:40.350835 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:40.504531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:40.503425 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:41.487903 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:41.487875 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:43.148530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:43.145547 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:44.771728 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:44.771690 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:46.385527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:46.377556 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:46.916207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:46.916175 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:47.641027 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:47.640991 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:48.038993 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.038964 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf"] Oct 30 18:36:48.039995 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.039965 4603 cpu_manager.go:414] "containerMap: removing container" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" Oct 30 18:36:48.039995 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.039992 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" Oct 30 18:36:48.040214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.040014 4603 cpu_manager.go:414] "containerMap: removing container" podUID="67142e99-ffa0-4b21-8f93-735d5663c87e" containerName="istio-proxy" Oct 30 18:36:48.040214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.040023 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67142e99-ffa0-4b21-8f93-735d5663c87e" containerName="istio-proxy" Oct 30 18:36:48.040214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.040035 4603 cpu_manager.go:414] "containerMap: removing container" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" Oct 30 18:36:48.040214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.040043 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" Oct 30 18:36:48.040214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.040054 4603 cpu_manager.go:414] "containerMap: removing container" podUID="67142e99-ffa0-4b21-8f93-735d5663c87e" containerName="istio-proxy" Oct 30 18:36:48.040214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.040059 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67142e99-ffa0-4b21-8f93-735d5663c87e" containerName="istio-proxy" Oct 30 18:36:48.040214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.040178 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a35d239-64fb-4752-91e9-10f61d87713f" containerName="main" Oct 30 18:36:48.040214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.040187 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="67142e99-ffa0-4b21-8f93-735d5663c87e" containerName="istio-proxy" Oct 30 18:36:48.040214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.040194 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc4f5be5-e07f-4960-ace5-4023198470be" containerName="main" Oct 30 18:36:48.040214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.040203 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="67142e99-ffa0-4b21-8f93-735d5663c87e" containerName="istio-proxy" Oct 30 18:36:48.045892 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.045868 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.066647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.066624 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-58d08b-openshift-default-dockercfg-f86xr\"" Oct 30 18:36:48.083286 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.083246 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf"] Oct 30 18:36:48.137225 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.137190 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:48.215525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.208835 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-credential-socket\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.215525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.208932 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-workload-certs\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.215525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.208969 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istiod-ca-cert\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.215525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.209121 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-podinfo\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.215525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.209216 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-data\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.215525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.209320 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-token\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.215525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.209417 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-envoy\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.243528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.234914 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvcx4\" (UniqueName: \"kubernetes.io/projected/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-kube-api-access-hvcx4\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.243528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.235045 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-workload-socket\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.299576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.297822 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:48.344523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.337006 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-podinfo\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.344523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.337730 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-data\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.344523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.337875 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-token\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.344523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.338324 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-data\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.383318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.346974 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-envoy\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.383318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.347057 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvcx4\" (UniqueName: \"kubernetes.io/projected/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-kube-api-access-hvcx4\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.383318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.347133 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-workload-socket\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.383318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.347218 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-credential-socket\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.383318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.347238 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-workload-certs\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.383318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.347261 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istiod-ca-cert\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.383318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.348060 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istiod-ca-cert\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.383318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.360829 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-workload-socket\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.383318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.361385 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-credential-socket\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.383318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.361653 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-workload-certs\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.406533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.396267 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-token\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.406533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.396593 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-envoy\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.414753 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.411409 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-podinfo\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.415685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.415622 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvcx4\" (UniqueName: \"kubernetes.io/projected/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-kube-api-access-hvcx4\") pod \"e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:48.461957 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.461926 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:48.676586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:48.674113 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:49.024554 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:36:49.024482 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5314716_efc8_4ce5_843f_fb4f5f2ba7e3.slice/crio-373ee226b0ce77bd751382b6939f3f87fe1d7dbc92ea737a64f7137dfdce78ca WatchSource:0}: Error finding container 373ee226b0ce77bd751382b6939f3f87fe1d7dbc92ea737a64f7137dfdce78ca: Status 404 returned error can't find the container with id 373ee226b0ce77bd751382b6939f3f87fe1d7dbc92ea737a64f7137dfdce78ca Oct 30 18:36:49.025551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:49.025524 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf"] Oct 30 18:36:49.027295 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:49.027249 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:36:49.027367 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:49.027329 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:36:49.027367 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:49.027358 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:36:49.312383 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:49.312299 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" event={"ID":"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3","Type":"ContainerStarted","Data":"290e03c48d5b4e3be632e3fdaaa0e1f2be7841d7d786197ac96e93474e3cfe88"} Oct 30 18:36:49.312383 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:49.312360 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" event={"ID":"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3","Type":"ContainerStarted","Data":"373ee226b0ce77bd751382b6939f3f87fe1d7dbc92ea737a64f7137dfdce78ca"} Oct 30 18:36:49.359338 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:49.359290 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" podStartSLOduration=1.359272904 podStartE2EDuration="1.359272904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:49.356686528 +0000 UTC m=+2670.072256437" watchObservedRunningTime="2026-10-30 18:36:49.359272904 +0000 UTC m=+2670.074842778" Oct 30 18:36:49.674460 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:49.674384 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:50.073299 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:50.073256 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:50.681100 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:50.681057 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:51.325965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:51.325932 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:51.326948 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:51.326928 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:36:51.531664 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:51.531626 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:52.032538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:52.023117 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:52.202525 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:52.200429 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:52.312575 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:36:52.311582 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:36:52.419060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:52.419022 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:52.568393 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:52.568299 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:52.743378 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:52.740377 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:53.088147 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:53.088117 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:53.731957 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:53.731927 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:54.053527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:54.043952 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:54.783273 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:54.783232 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb"] Oct 30 18:36:54.859322 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:54.853851 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:55.014560 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:55.014527 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:55.481283 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:55.481245 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:55.964059 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:55.964029 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:56.144625 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:56.143521 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:56.913814 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:56.913777 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:58.117031 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:58.116997 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:58.280524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:58.280188 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:58.992569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:36:58.987969 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:00.000934 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:00.000899 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:00.156212 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:00.156175 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:00.360934 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:00.360900 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:00.391285 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:00.391253 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4"] Oct 30 18:37:00.391570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:00.391480 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" podUID="e92796f5-3580-47ed-bcde-da5fb7075252" containerName="istio-proxy" containerID="cri-o://3b374f6038729b8f36db585561f52e7a6e2081fa6224d206960b00ac8934c2af" gracePeriod=30 Oct 30 18:37:00.403364 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:00.403334 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv"] Oct 30 18:37:00.403587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:00.403551 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" podUID="a2d95ad6-c697-48a0-8592-0ff95ce3767b" containerName="istio-proxy" containerID="cri-o://24d8106b8305624ad78d84f82882a972234217dd577d3d34c1d6aa6f65ac9af1" gracePeriod=30 Oct 30 18:37:00.573542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:00.566748 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:01.997389 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:01.997349 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:02.141627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:02.141600 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:02.326544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:02.326263 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:04.142329 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:04.142264 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:05.656815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:05.656762 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:05.830932 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:05.830900 4603 generic.go:356] "Generic (PLEG): container finished" podID="a2d95ad6-c697-48a0-8592-0ff95ce3767b" containerID="24d8106b8305624ad78d84f82882a972234217dd577d3d34c1d6aa6f65ac9af1" exitCode=0 Oct 30 18:37:05.831092 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:05.830931 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" event={"ID":"a2d95ad6-c697-48a0-8592-0ff95ce3767b","Type":"ContainerDied","Data":"24d8106b8305624ad78d84f82882a972234217dd577d3d34c1d6aa6f65ac9af1"} Oct 30 18:37:05.831092 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:05.830994 4603 scope.go:122] "RemoveContainer" containerID="69c0db244afc76ed20a51a416413b30baf9462d999fb95e2b0b5f000b1f60a21" Oct 30 18:37:05.833043 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:05.833016 4603 generic.go:356] "Generic (PLEG): container finished" podID="e92796f5-3580-47ed-bcde-da5fb7075252" containerID="3b374f6038729b8f36db585561f52e7a6e2081fa6224d206960b00ac8934c2af" exitCode=0 Oct 30 18:37:05.833191 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:05.833092 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" event={"ID":"e92796f5-3580-47ed-bcde-da5fb7075252","Type":"ContainerDied","Data":"3b374f6038729b8f36db585561f52e7a6e2081fa6224d206960b00ac8934c2af"} Oct 30 18:37:05.833191 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:05.833142 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" event={"ID":"e92796f5-3580-47ed-bcde-da5fb7075252","Type":"ContainerDied","Data":"2b9f3bcac4336763c47254b83ce901ca220c5322bbdad7a27984082bc94f0be2"} Oct 30 18:37:05.833191 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:05.833160 4603 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b9f3bcac4336763c47254b83ce901ca220c5322bbdad7a27984082bc94f0be2" Oct 30 18:37:05.844485 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:05.844464 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:37:05.878309 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:05.878283 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:37:06.000559 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.000530 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-workload-certs\") pod \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " Oct 30 18:37:06.000559 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.000572 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e92796f5-3580-47ed-bcde-da5fb7075252-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92796f5-3580-47ed-bcde-da5fb7075252-istiod-ca-cert\") pod \"e92796f5-3580-47ed-bcde-da5fb7075252\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " Oct 30 18:37:06.000825 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.000648 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-token\" (UniqueName: \"kubernetes.io/projected/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-token\") pod \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " Oct 30 18:37:06.000825 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.000711 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e92796f5-3580-47ed-bcde-da5fb7075252-istio-token\" (UniqueName: \"kubernetes.io/projected/e92796f5-3580-47ed-bcde-da5fb7075252-istio-token\") pod \"e92796f5-3580-47ed-bcde-da5fb7075252\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " Oct 30 18:37:06.000825 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.000752 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-istio-data\") pod \"e92796f5-3580-47ed-bcde-da5fb7075252\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " Oct 30 18:37:06.000825 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.000779 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istiod-ca-cert\") pod \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " Oct 30 18:37:06.001041 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.000852 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-workload-certs" pod "a2d95ad6-c697-48a0-8592-0ff95ce3767b" (UID: "a2d95ad6-c697-48a0-8592-0ff95ce3767b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:06.001041 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.000961 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92796f5-3580-47ed-bcde-da5fb7075252-istiod-ca-cert" pod "e92796f5-3580-47ed-bcde-da5fb7075252" (UID: "e92796f5-3580-47ed-bcde-da5fb7075252"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:06.001567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.001313 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istiod-ca-cert" pod "a2d95ad6-c697-48a0-8592-0ff95ce3767b" (UID: "a2d95ad6-c697-48a0-8592-0ff95ce3767b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:06.001567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.001321 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-istio-data" pod "e92796f5-3580-47ed-bcde-da5fb7075252" (UID: "e92796f5-3580-47ed-bcde-da5fb7075252"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:06.001567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.000849 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-workload-certs\") pod \"e92796f5-3580-47ed-bcde-da5fb7075252\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " Oct 30 18:37:06.001567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.001337 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-workload-certs" pod "e92796f5-3580-47ed-bcde-da5fb7075252" (UID: "e92796f5-3580-47ed-bcde-da5fb7075252"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:06.001567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.001391 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-workload-socket\") pod \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " Oct 30 18:37:06.001567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.001460 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-data\") pod \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " Oct 30 18:37:06.001961 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.001781 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-data" pod "a2d95ad6-c697-48a0-8592-0ff95ce3767b" (UID: "a2d95ad6-c697-48a0-8592-0ff95ce3767b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:06.001961 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.001848 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-workload-socket" pod "a2d95ad6-c697-48a0-8592-0ff95ce3767b" (UID: "a2d95ad6-c697-48a0-8592-0ff95ce3767b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:06.001961 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.001493 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-credential-socket\") pod \"e92796f5-3580-47ed-bcde-da5fb7075252\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " Oct 30 18:37:06.002144 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.001975 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-envoy\") pod \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " Oct 30 18:37:06.002144 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.001987 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-credential-socket" pod "e92796f5-3580-47ed-bcde-da5fb7075252" (UID: "e92796f5-3580-47ed-bcde-da5fb7075252"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:06.002144 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.002064 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e92796f5-3580-47ed-bcde-da5fb7075252-kube-api-access-f88g2\" (UniqueName: \"kubernetes.io/projected/e92796f5-3580-47ed-bcde-da5fb7075252-kube-api-access-f88g2\") pod \"e92796f5-3580-47ed-bcde-da5fb7075252\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " Oct 30 18:37:06.002144 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.002106 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-workload-socket\") pod \"e92796f5-3580-47ed-bcde-da5fb7075252\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " Oct 30 18:37:06.002358 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.002188 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e92796f5-3580-47ed-bcde-da5fb7075252-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92796f5-3580-47ed-bcde-da5fb7075252-istio-podinfo\") pod \"e92796f5-3580-47ed-bcde-da5fb7075252\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " Oct 30 18:37:06.002358 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.002220 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-podinfo\") pod \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " Oct 30 18:37:06.002789 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.002756 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-workload-socket" pod "e92796f5-3580-47ed-bcde-da5fb7075252" (UID: "e92796f5-3580-47ed-bcde-da5fb7075252"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:06.003645 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.003620 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-credential-socket" pod "a2d95ad6-c697-48a0-8592-0ff95ce3767b" (UID: "a2d95ad6-c697-48a0-8592-0ff95ce3767b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:06.003772 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.003659 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-credential-socket\") pod \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " Oct 30 18:37:06.003772 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.003727 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2d95ad6-c697-48a0-8592-0ff95ce3767b-kube-api-access-9h4v4\" (UniqueName: \"kubernetes.io/projected/a2d95ad6-c697-48a0-8592-0ff95ce3767b-kube-api-access-9h4v4\") pod \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\" (UID: \"a2d95ad6-c697-48a0-8592-0ff95ce3767b\") " Oct 30 18:37:06.004959 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.004928 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-token" pod "a2d95ad6-c697-48a0-8592-0ff95ce3767b" (UID: "a2d95ad6-c697-48a0-8592-0ff95ce3767b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:06.005059 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.004966 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92796f5-3580-47ed-bcde-da5fb7075252-istio-token" pod "e92796f5-3580-47ed-bcde-da5fb7075252" (UID: "e92796f5-3580-47ed-bcde-da5fb7075252"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:06.005059 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.004977 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-istio-envoy\") pod \"e92796f5-3580-47ed-bcde-da5fb7075252\" (UID: \"e92796f5-3580-47ed-bcde-da5fb7075252\") " Oct 30 18:37:06.006061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.005872 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.006061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.005901 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.006061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.005916 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92796f5-3580-47ed-bcde-da5fb7075252-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.006061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.005928 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.006061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.005941 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92796f5-3580-47ed-bcde-da5fb7075252-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.006061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.005953 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.006061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.005965 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.006061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.005979 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.006061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.005992 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.006061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.006004 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.006061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.006018 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.006061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.006030 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.007118 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.007092 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-envoy" pod "a2d95ad6-c697-48a0-8592-0ff95ce3767b" (UID: "a2d95ad6-c697-48a0-8592-0ff95ce3767b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:06.007207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.007122 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92796f5-3580-47ed-bcde-da5fb7075252-kube-api-access-f88g2" pod "e92796f5-3580-47ed-bcde-da5fb7075252" (UID: "e92796f5-3580-47ed-bcde-da5fb7075252"). InnerVolumeSpecName "kube-api-access-f88g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:06.008313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.008285 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-podinfo" pod "a2d95ad6-c697-48a0-8592-0ff95ce3767b" (UID: "a2d95ad6-c697-48a0-8592-0ff95ce3767b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:06.008313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.008306 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e92796f5-3580-47ed-bcde-da5fb7075252-istio-podinfo" pod "e92796f5-3580-47ed-bcde-da5fb7075252" (UID: "e92796f5-3580-47ed-bcde-da5fb7075252"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:06.008451 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.008378 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-istio-envoy" pod "e92796f5-3580-47ed-bcde-da5fb7075252" (UID: "e92796f5-3580-47ed-bcde-da5fb7075252"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:06.009007 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.008991 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d95ad6-c697-48a0-8592-0ff95ce3767b-kube-api-access-9h4v4" pod "a2d95ad6-c697-48a0-8592-0ff95ce3767b" (UID: "a2d95ad6-c697-48a0-8592-0ff95ce3767b"). InnerVolumeSpecName "kube-api-access-9h4v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:06.107140 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.107074 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f88g2\" (UniqueName: \"kubernetes.io/projected/e92796f5-3580-47ed-bcde-da5fb7075252-kube-api-access-f88g2\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.107140 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.107100 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92796f5-3580-47ed-bcde-da5fb7075252-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.107140 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.107115 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.107140 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.107128 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h4v4\" (UniqueName: \"kubernetes.io/projected/a2d95ad6-c697-48a0-8592-0ff95ce3767b-kube-api-access-9h4v4\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.107140 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.107140 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92796f5-3580-47ed-bcde-da5fb7075252-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.107495 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.107153 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2d95ad6-c697-48a0-8592-0ff95ce3767b-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.841397 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.841365 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" Oct 30 18:37:06.841835 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.841417 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv" event={"ID":"a2d95ad6-c697-48a0-8592-0ff95ce3767b","Type":"ContainerDied","Data":"a0a112f2f6b1cd937582f84d851d5724311e1e73c92c8d5d9a36dcf22db76d68"} Oct 30 18:37:06.841835 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.841472 4603 scope.go:122] "RemoveContainer" containerID="24d8106b8305624ad78d84f82882a972234217dd577d3d34c1d6aa6f65ac9af1" Oct 30 18:37:06.841835 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.841554 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4" Oct 30 18:37:06.921313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.921289 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4"] Oct 30 18:37:06.932671 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.932653 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-28nm4"] Oct 30 18:37:06.957971 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.957951 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv"] Oct 30 18:37:06.962080 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:06.962057 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-485pv"] Oct 30 18:37:07.248673 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:07.248593 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:07.411636 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:07.411607 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2d95ad6-c697-48a0-8592-0ff95ce3767b" path="/var/lib/kubelet/pods/a2d95ad6-c697-48a0-8592-0ff95ce3767b/volumes" Oct 30 18:37:07.412207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:07.412192 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e92796f5-3580-47ed-bcde-da5fb7075252" path="/var/lib/kubelet/pods/e92796f5-3580-47ed-bcde-da5fb7075252/volumes" Oct 30 18:37:08.757018 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:08.756976 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:09.051580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:09.041921 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:10.061123 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.061088 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" containerID="cri-o://b9aa92a5eb0ef6c1c056b9298aaab7c07391e5fc8c72452a956c6ad34dfe8d98" gracePeriod=45 Oct 30 18:37:10.348183 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.348158 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:37:10.379552 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.379523 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-dshm\") pod \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " Oct 30 18:37:10.379674 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.379589 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-tls-certs\" (UniqueName: \"kubernetes.io/secret/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-tls-certs\") pod \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " Oct 30 18:37:10.379674 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.379639 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-kube-api-access-dgswl\" (UniqueName: \"kubernetes.io/projected/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-kube-api-access-dgswl\") pod \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " Oct 30 18:37:10.379805 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.379734 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-home\" (UniqueName: \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-home\") pod \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " Oct 30 18:37:10.379871 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.379815 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-tmp-dir\") pod \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " Oct 30 18:37:10.379984 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.379958 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-model-cache\") pod \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\" (UID: \"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b\") " Oct 30 18:37:10.380070 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.380049 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-home" pod "dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" (UID: "dcdfbbbe-62bd-44c0-806b-d8bdbc50729b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:10.380132 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.380071 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-tmp-dir" pod "dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" (UID: "dcdfbbbe-62bd-44c0-806b-d8bdbc50729b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:10.380298 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.380209 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-model-cache" pod "dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" (UID: "dcdfbbbe-62bd-44c0-806b-d8bdbc50729b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:10.381221 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.381197 4603 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-home\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:10.381309 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.381227 4603 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-tmp-dir\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:10.381309 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.381244 4603 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-model-cache\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:10.382012 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.381983 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-dshm" pod "dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" (UID: "dcdfbbbe-62bd-44c0-806b-d8bdbc50729b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:10.382117 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.382062 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-tls-certs" pod "dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" (UID: "dcdfbbbe-62bd-44c0-806b-d8bdbc50729b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:10.382162 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.382122 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-kube-api-access-dgswl" pod "dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" (UID: "dcdfbbbe-62bd-44c0-806b-d8bdbc50729b"). InnerVolumeSpecName "kube-api-access-dgswl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:10.483268 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.483232 4603 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-dshm\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:10.483268 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.483267 4603 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-tls-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:10.483461 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.483285 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgswl\" (UniqueName: \"kubernetes.io/projected/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b-kube-api-access-dgswl\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:10.882527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.882315 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:10.942242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.942207 4603 generic.go:356] "Generic (PLEG): container finished" podID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerID="b9aa92a5eb0ef6c1c056b9298aaab7c07391e5fc8c72452a956c6ad34dfe8d98" exitCode=0 Oct 30 18:37:10.942537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.942493 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" event={"ID":"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b","Type":"ContainerDied","Data":"b9aa92a5eb0ef6c1c056b9298aaab7c07391e5fc8c72452a956c6ad34dfe8d98"} Oct 30 18:37:10.942686 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.942670 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" event={"ID":"dcdfbbbe-62bd-44c0-806b-d8bdbc50729b","Type":"ContainerDied","Data":"4fbf06616d8af6d381f0946c4b3da270578be790f518e2f6de6cad4ad5f75734"} Oct 30 18:37:10.942781 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.942764 4603 scope.go:122] "RemoveContainer" containerID="b9aa92a5eb0ef6c1c056b9298aaab7c07391e5fc8c72452a956c6ad34dfe8d98" Oct 30 18:37:10.942959 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.942933 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb" Oct 30 18:37:10.964302 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.964276 4603 scope.go:122] "RemoveContainer" containerID="b9aa92a5eb0ef6c1c056b9298aaab7c07391e5fc8c72452a956c6ad34dfe8d98" Oct 30 18:37:10.964762 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:37:10.964737 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9aa92a5eb0ef6c1c056b9298aaab7c07391e5fc8c72452a956c6ad34dfe8d98\": container with ID starting with b9aa92a5eb0ef6c1c056b9298aaab7c07391e5fc8c72452a956c6ad34dfe8d98 not found: ID does not exist" containerID="b9aa92a5eb0ef6c1c056b9298aaab7c07391e5fc8c72452a956c6ad34dfe8d98" Oct 30 18:37:10.964880 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:10.964770 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9aa92a5eb0ef6c1c056b9298aaab7c07391e5fc8c72452a956c6ad34dfe8d98"} err="failed to get container status \"b9aa92a5eb0ef6c1c056b9298aaab7c07391e5fc8c72452a956c6ad34dfe8d98\": rpc error: code = NotFound desc = could not find container \"b9aa92a5eb0ef6c1c056b9298aaab7c07391e5fc8c72452a956c6ad34dfe8d98\": container with ID starting with b9aa92a5eb0ef6c1c056b9298aaab7c07391e5fc8c72452a956c6ad34dfe8d98 not found: ID does not exist" Oct 30 18:37:11.015671 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:11.015638 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb"] Oct 30 18:37:11.027189 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:11.027167 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c8eef3/auth-test-model-c8eef3-kserve-6f874bb9bf-cx7bb"] Oct 30 18:37:11.411247 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:11.411218 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" path="/var/lib/kubelet/pods/dcdfbbbe-62bd-44c0-806b-d8bdbc50729b/volumes" Oct 30 18:37:11.543675 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:11.543643 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf"] Oct 30 18:37:11.580437 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:11.580409 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc"] Oct 30 18:37:11.723427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:11.723341 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:11.886103 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:11.886066 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:12.853829 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:12.853787 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:14.618533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:14.617244 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:16.074293 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.074265 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:16.249740 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.249700 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:16.418696 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.415087 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:16.543926 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.543887 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf" podUID="cb9b49ca-d057-4d90-bab8-1519401aa457" containerName="payload-processing" containerID="cri-o://45bda4e54b224495b141147eac4e025ed12cee8d9261269659d42c2ab644975f" gracePeriod=25 Oct 30 18:37:16.580899 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.580871 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc" podUID="103f4661-1a62-4f4d-8089-3ceabd3271ad" containerName="payload-pre-processing" containerID="cri-o://da804c96d6cd9673a8e34be3b42b5c9647cd5db7f594eabbee1ab4d353bcc31c" gracePeriod=25 Oct 30 18:37:16.659794 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.659764 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl"] Oct 30 18:37:16.660843 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.660817 4603 cpu_manager.go:414] "containerMap: removing container" podUID="a2d95ad6-c697-48a0-8592-0ff95ce3767b" containerName="istio-proxy" Oct 30 18:37:16.660843 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.660843 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2d95ad6-c697-48a0-8592-0ff95ce3767b" containerName="istio-proxy" Oct 30 18:37:16.661056 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.660873 4603 cpu_manager.go:414] "containerMap: removing container" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" Oct 30 18:37:16.661056 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.660881 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" Oct 30 18:37:16.661056 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.660894 4603 cpu_manager.go:414] "containerMap: removing container" podUID="e92796f5-3580-47ed-bcde-da5fb7075252" containerName="istio-proxy" Oct 30 18:37:16.661056 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.660903 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e92796f5-3580-47ed-bcde-da5fb7075252" containerName="istio-proxy" Oct 30 18:37:16.661192 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.661115 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2d95ad6-c697-48a0-8592-0ff95ce3767b" containerName="istio-proxy" Oct 30 18:37:16.661192 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.661137 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2d95ad6-c697-48a0-8592-0ff95ce3767b" containerName="istio-proxy" Oct 30 18:37:16.661192 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.661149 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcdfbbbe-62bd-44c0-806b-d8bdbc50729b" containerName="main" Oct 30 18:37:16.661192 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.661157 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="e92796f5-3580-47ed-bcde-da5fb7075252" containerName="istio-proxy" Oct 30 18:37:16.665392 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.665371 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:16.681859 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.681825 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-ac422f\"/\"kube-root-ca.crt\"" Oct 30 18:37:16.687182 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.687161 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl"] Oct 30 18:37:16.694023 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.693669 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-ac422f\"/\"praxis-route-ac422f-kserve-self-signed-certs\"" Oct 30 18:37:16.694023 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.693804 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-ac422f\"/\"default-dockercfg-rwbz9\"" Oct 30 18:37:16.694023 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.693894 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-ac422f\"/\"openshift-service-ca.crt\"" Oct 30 18:37:16.749204 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.749173 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-dshm\") pod \"praxis-route-ac422f-kserve-85c95f9b9-djkpl\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:16.749379 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.749268 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gqvs\" (UniqueName: \"kubernetes.io/projected/81d31061-bdab-430c-9dba-d50879b97a29-kube-api-access-8gqvs\") pod \"praxis-route-ac422f-kserve-85c95f9b9-djkpl\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:16.749379 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.749290 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-model-cache\") pod \"praxis-route-ac422f-kserve-85c95f9b9-djkpl\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:16.749379 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.749323 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-tmp-dir\") pod \"praxis-route-ac422f-kserve-85c95f9b9-djkpl\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:16.749379 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.749367 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-home\") pod \"praxis-route-ac422f-kserve-85c95f9b9-djkpl\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:16.749576 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.749555 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d31061-bdab-430c-9dba-d50879b97a29-tls-certs\") pod \"praxis-route-ac422f-kserve-85c95f9b9-djkpl\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:16.759697 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.759574 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:16.851159 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.851132 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d31061-bdab-430c-9dba-d50879b97a29-tls-certs\") pod \"praxis-route-ac422f-kserve-85c95f9b9-djkpl\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:16.851329 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.851229 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-dshm\") pod \"praxis-route-ac422f-kserve-85c95f9b9-djkpl\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:16.851329 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.851320 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gqvs\" (UniqueName: \"kubernetes.io/projected/81d31061-bdab-430c-9dba-d50879b97a29-kube-api-access-8gqvs\") pod \"praxis-route-ac422f-kserve-85c95f9b9-djkpl\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:16.851456 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.851348 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-model-cache\") pod \"praxis-route-ac422f-kserve-85c95f9b9-djkpl\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:16.851456 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.851393 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-tmp-dir\") pod \"praxis-route-ac422f-kserve-85c95f9b9-djkpl\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:16.851597 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.851492 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-home\") pod \"praxis-route-ac422f-kserve-85c95f9b9-djkpl\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:16.852594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.852038 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-home\") pod \"praxis-route-ac422f-kserve-85c95f9b9-djkpl\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:16.852594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.852297 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-model-cache\") pod \"praxis-route-ac422f-kserve-85c95f9b9-djkpl\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:16.852594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.852557 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-tmp-dir\") pod \"praxis-route-ac422f-kserve-85c95f9b9-djkpl\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:16.856598 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.855199 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d31061-bdab-430c-9dba-d50879b97a29-tls-certs\") pod \"praxis-route-ac422f-kserve-85c95f9b9-djkpl\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:16.856598 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.856248 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-dshm\") pod \"praxis-route-ac422f-kserve-85c95f9b9-djkpl\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:16.875123 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.875099 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gqvs\" (UniqueName: \"kubernetes.io/projected/81d31061-bdab-430c-9dba-d50879b97a29-kube-api-access-8gqvs\") pod \"praxis-route-ac422f-kserve-85c95f9b9-djkpl\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:16.926543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:16.919721 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf" Oct 30 18:37:17.086597 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.004853 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:17.137574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.108343 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb9b49ca-d057-4d90-bab8-1519401aa457-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9b49ca-d057-4d90-bab8-1519401aa457-plugins-config-volume\") pod \"cb9b49ca-d057-4d90-bab8-1519401aa457\" (UID: \"cb9b49ca-d057-4d90-bab8-1519401aa457\") " Oct 30 18:37:17.137574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.108431 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb9b49ca-d057-4d90-bab8-1519401aa457-kube-api-access-rcl2j\" (UniqueName: \"kubernetes.io/projected/cb9b49ca-d057-4d90-bab8-1519401aa457-kube-api-access-rcl2j\") pod \"cb9b49ca-d057-4d90-bab8-1519401aa457\" (UID: \"cb9b49ca-d057-4d90-bab8-1519401aa457\") " Oct 30 18:37:17.137574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.111019 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb9b49ca-d057-4d90-bab8-1519401aa457-plugins-config-volume" pod "cb9b49ca-d057-4d90-bab8-1519401aa457" (UID: "cb9b49ca-d057-4d90-bab8-1519401aa457"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:17.169037 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.169000 4603 generic.go:356] "Generic (PLEG): container finished" podID="103f4661-1a62-4f4d-8089-3ceabd3271ad" containerID="da804c96d6cd9673a8e34be3b42b5c9647cd5db7f594eabbee1ab4d353bcc31c" exitCode=0 Oct 30 18:37:17.169390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.169360 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc" event={"ID":"103f4661-1a62-4f4d-8089-3ceabd3271ad","Type":"ContainerDied","Data":"da804c96d6cd9673a8e34be3b42b5c9647cd5db7f594eabbee1ab4d353bcc31c"} Oct 30 18:37:17.171216 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.171190 4603 generic.go:356] "Generic (PLEG): container finished" podID="cb9b49ca-d057-4d90-bab8-1519401aa457" containerID="45bda4e54b224495b141147eac4e025ed12cee8d9261269659d42c2ab644975f" exitCode=0 Oct 30 18:37:17.171443 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.171424 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf" event={"ID":"cb9b49ca-d057-4d90-bab8-1519401aa457","Type":"ContainerDied","Data":"45bda4e54b224495b141147eac4e025ed12cee8d9261269659d42c2ab644975f"} Oct 30 18:37:17.171619 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.171600 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf" event={"ID":"cb9b49ca-d057-4d90-bab8-1519401aa457","Type":"ContainerDied","Data":"0f1f64c78cdc74497d966ac2d2c5c531c0fb43b8903976d55b19b4adc2a1946d"} Oct 30 18:37:17.171750 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.171735 4603 scope.go:122] "RemoveContainer" containerID="45bda4e54b224495b141147eac4e025ed12cee8d9261269659d42c2ab644975f" Oct 30 18:37:17.172030 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.172010 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf" Oct 30 18:37:17.193998 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.193958 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb9b49ca-d057-4d90-bab8-1519401aa457-kube-api-access-rcl2j" pod "cb9b49ca-d057-4d90-bab8-1519401aa457" (UID: "cb9b49ca-d057-4d90-bab8-1519401aa457"). InnerVolumeSpecName "kube-api-access-rcl2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:17.201033 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.201013 4603 scope.go:122] "RemoveContainer" containerID="45bda4e54b224495b141147eac4e025ed12cee8d9261269659d42c2ab644975f" Oct 30 18:37:17.201543 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:37:17.201468 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45bda4e54b224495b141147eac4e025ed12cee8d9261269659d42c2ab644975f\": container with ID starting with 45bda4e54b224495b141147eac4e025ed12cee8d9261269659d42c2ab644975f not found: ID does not exist" containerID="45bda4e54b224495b141147eac4e025ed12cee8d9261269659d42c2ab644975f" Oct 30 18:37:17.201543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.201519 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45bda4e54b224495b141147eac4e025ed12cee8d9261269659d42c2ab644975f"} err="failed to get container status \"45bda4e54b224495b141147eac4e025ed12cee8d9261269659d42c2ab644975f\": rpc error: code = NotFound desc = could not find container \"45bda4e54b224495b141147eac4e025ed12cee8d9261269659d42c2ab644975f\": container with ID starting with 45bda4e54b224495b141147eac4e025ed12cee8d9261269659d42c2ab644975f not found: ID does not exist" Oct 30 18:37:17.207314 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.207289 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc" Oct 30 18:37:17.211225 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.211199 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9b49ca-d057-4d90-bab8-1519401aa457-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:17.211328 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.211228 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcl2j\" (UniqueName: \"kubernetes.io/projected/cb9b49ca-d057-4d90-bab8-1519401aa457-kube-api-access-rcl2j\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:17.311775 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.311739 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/103f4661-1a62-4f4d-8089-3ceabd3271ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103f4661-1a62-4f4d-8089-3ceabd3271ad-plugins-config-volume\") pod \"103f4661-1a62-4f4d-8089-3ceabd3271ad\" (UID: \"103f4661-1a62-4f4d-8089-3ceabd3271ad\") " Oct 30 18:37:17.312180 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.312149 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103f4661-1a62-4f4d-8089-3ceabd3271ad-kube-api-access-9hgp4\" (UniqueName: \"kubernetes.io/projected/103f4661-1a62-4f4d-8089-3ceabd3271ad-kube-api-access-9hgp4\") pod \"103f4661-1a62-4f4d-8089-3ceabd3271ad\" (UID: \"103f4661-1a62-4f4d-8089-3ceabd3271ad\") " Oct 30 18:37:17.312319 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.312159 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103f4661-1a62-4f4d-8089-3ceabd3271ad-plugins-config-volume" pod "103f4661-1a62-4f4d-8089-3ceabd3271ad" (UID: "103f4661-1a62-4f4d-8089-3ceabd3271ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:17.312818 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.312797 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103f4661-1a62-4f4d-8089-3ceabd3271ad-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:17.314920 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.314900 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103f4661-1a62-4f4d-8089-3ceabd3271ad-kube-api-access-9hgp4" pod "103f4661-1a62-4f4d-8089-3ceabd3271ad" (UID: "103f4661-1a62-4f4d-8089-3ceabd3271ad"). InnerVolumeSpecName "kube-api-access-9hgp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:17.413746 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.413678 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hgp4\" (UniqueName: \"kubernetes.io/projected/103f4661-1a62-4f4d-8089-3ceabd3271ad-kube-api-access-9hgp4\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:17.502565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.502541 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl"] Oct 30 18:37:17.502900 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:37:17.502880 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81d31061_bdab_430c_9dba_d50879b97a29.slice/crio-451df865a8a92cb9d12ecdca9cefbf2d0774af303bc229a64315b4d4918ca267 WatchSource:0}: Error finding container 451df865a8a92cb9d12ecdca9cefbf2d0774af303bc229a64315b4d4918ca267: Status 404 returned error can't find the container with id 451df865a8a92cb9d12ecdca9cefbf2d0774af303bc229a64315b4d4918ca267 Oct 30 18:37:17.522912 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.522886 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf"] Oct 30 18:37:17.528266 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.528245 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b00d69-5b6cd74cb4-fs4rf"] Oct 30 18:37:17.999901 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:17.999869 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w"] Oct 30 18:37:18.000579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.000560 4603 cpu_manager.go:414] "containerMap: removing container" podUID="cb9b49ca-d057-4d90-bab8-1519401aa457" containerName="payload-processing" Oct 30 18:37:18.000579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.000579 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb9b49ca-d057-4d90-bab8-1519401aa457" containerName="payload-processing" Oct 30 18:37:18.000757 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.000616 4603 cpu_manager.go:414] "containerMap: removing container" podUID="103f4661-1a62-4f4d-8089-3ceabd3271ad" containerName="payload-pre-processing" Oct 30 18:37:18.000757 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.000622 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103f4661-1a62-4f4d-8089-3ceabd3271ad" containerName="payload-pre-processing" Oct 30 18:37:18.000757 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.000638 4603 cpu_manager.go:414] "containerMap: removing container" podUID="a2d95ad6-c697-48a0-8592-0ff95ce3767b" containerName="istio-proxy" Oct 30 18:37:18.000757 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.000643 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2d95ad6-c697-48a0-8592-0ff95ce3767b" containerName="istio-proxy" Oct 30 18:37:18.000757 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.000749 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb9b49ca-d057-4d90-bab8-1519401aa457" containerName="payload-processing" Oct 30 18:37:18.000957 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.000765 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="103f4661-1a62-4f4d-8089-3ceabd3271ad" containerName="payload-pre-processing" Oct 30 18:37:18.004469 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.004450 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" Oct 30 18:37:18.026471 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.026444 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-58d08b-serving-cert\"" Oct 30 18:37:18.033434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.033402 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w"] Oct 30 18:37:18.123399 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.123367 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79321de-091c-47e7-8320-3471ef8f554b-openshift-service-ca\") pod \"maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w\" (UID: \"d79321de-091c-47e7-8320-3471ef8f554b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" Oct 30 18:37:18.123872 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.123403 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79321de-091c-47e7-8320-3471ef8f554b-metrics-certs\") pod \"maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w\" (UID: \"d79321de-091c-47e7-8320-3471ef8f554b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" Oct 30 18:37:18.123872 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.123626 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lstz\" (UniqueName: \"kubernetes.io/projected/d79321de-091c-47e7-8320-3471ef8f554b-kube-api-access-2lstz\") pod \"maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w\" (UID: \"d79321de-091c-47e7-8320-3471ef8f554b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" Oct 30 18:37:18.123872 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.123775 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d79321de-091c-47e7-8320-3471ef8f554b-maas-api-tls\") pod \"maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w\" (UID: \"d79321de-091c-47e7-8320-3471ef8f554b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" Oct 30 18:37:18.127472 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.127441 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:18.185144 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.185103 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc" event={"ID":"103f4661-1a62-4f4d-8089-3ceabd3271ad","Type":"ContainerDied","Data":"fd8c23b3de991f13d97b41d43707c0a82f1bbf09b8c9fae70b469fa833b5ec53"} Oct 30 18:37:18.185316 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.185220 4603 scope.go:122] "RemoveContainer" containerID="da804c96d6cd9673a8e34be3b42b5c9647cd5db7f594eabbee1ab4d353bcc31c" Oct 30 18:37:18.185425 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.185407 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc" Oct 30 18:37:18.191539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.191481 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" event={"ID":"81d31061-bdab-430c-9dba-d50879b97a29","Type":"ContainerStarted","Data":"fd478adb11433e14e16d7e6d98139c62a19cb873c27e2cf3715a9d6e7ed1aead"} Oct 30 18:37:18.191539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.191535 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" event={"ID":"81d31061-bdab-430c-9dba-d50879b97a29","Type":"ContainerStarted","Data":"451df865a8a92cb9d12ecdca9cefbf2d0774af303bc229a64315b4d4918ca267"} Oct 30 18:37:18.258515 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.256806 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lstz\" (UniqueName: \"kubernetes.io/projected/d79321de-091c-47e7-8320-3471ef8f554b-kube-api-access-2lstz\") pod \"maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w\" (UID: \"d79321de-091c-47e7-8320-3471ef8f554b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" Oct 30 18:37:18.258515 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.256954 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d79321de-091c-47e7-8320-3471ef8f554b-maas-api-tls\") pod \"maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w\" (UID: \"d79321de-091c-47e7-8320-3471ef8f554b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" Oct 30 18:37:18.258515 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.257201 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79321de-091c-47e7-8320-3471ef8f554b-openshift-service-ca\") pod \"maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w\" (UID: \"d79321de-091c-47e7-8320-3471ef8f554b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" Oct 30 18:37:18.258515 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.257252 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79321de-091c-47e7-8320-3471ef8f554b-metrics-certs\") pod \"maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w\" (UID: \"d79321de-091c-47e7-8320-3471ef8f554b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" Oct 30 18:37:18.282529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.277542 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79321de-091c-47e7-8320-3471ef8f554b-openshift-service-ca\") pod \"maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w\" (UID: \"d79321de-091c-47e7-8320-3471ef8f554b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" Oct 30 18:37:18.304111 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.304076 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d79321de-091c-47e7-8320-3471ef8f554b-maas-api-tls\") pod \"maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w\" (UID: \"d79321de-091c-47e7-8320-3471ef8f554b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" Oct 30 18:37:18.318535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.307146 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79321de-091c-47e7-8320-3471ef8f554b-metrics-certs\") pod \"maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w\" (UID: \"d79321de-091c-47e7-8320-3471ef8f554b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" Oct 30 18:37:18.330564 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.330083 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:18.335107 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.335070 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc"] Oct 30 18:37:18.348831 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.348773 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lstz\" (UniqueName: \"kubernetes.io/projected/d79321de-091c-47e7-8320-3471ef8f554b-kube-api-access-2lstz\") pod \"maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w\" (UID: \"d79321de-091c-47e7-8320-3471ef8f554b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" Oct 30 18:37:18.357547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.350066 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b00d69-548865f679-gt8mc"] Oct 30 18:37:18.389906 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.386766 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" podStartSLOduration=2.386744037 podStartE2EDuration="2.386744037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:18.377980792 +0000 UTC m=+2699.093550691" watchObservedRunningTime="2026-10-30 18:37:18.386744037 +0000 UTC m=+2699.102313911" Oct 30 18:37:18.618780 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.618748 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" Oct 30 18:37:18.948910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.948882 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w"] Oct 30 18:37:18.952072 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:37:18.952032 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79321de_091c_47e7_8320_3471ef8f554b.slice/crio-02dd29294222bb11e0e173bf72d7d700549565cd00e7fc50010d9608725c1c70 WatchSource:0}: Error finding container 02dd29294222bb11e0e173bf72d7d700549565cd00e7fc50010d9608725c1c70: Status 404 returned error can't find the container with id 02dd29294222bb11e0e173bf72d7d700549565cd00e7fc50010d9608725c1c70 Oct 30 18:37:18.953694 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:18.953672 4603 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:37:19.015703 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:19.015678 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:19.354246 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:19.354207 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" event={"ID":"d79321de-091c-47e7-8320-3471ef8f554b","Type":"ContainerStarted","Data":"02dd29294222bb11e0e173bf72d7d700549565cd00e7fc50010d9608725c1c70"} Oct 30 18:37:19.415131 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:19.415104 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="103f4661-1a62-4f4d-8089-3ceabd3271ad" path="/var/lib/kubelet/pods/103f4661-1a62-4f4d-8089-3ceabd3271ad/volumes" Oct 30 18:37:19.415727 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:19.415709 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb9b49ca-d057-4d90-bab8-1519401aa457" path="/var/lib/kubelet/pods/cb9b49ca-d057-4d90-bab8-1519401aa457/volumes" Oct 30 18:37:20.027529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:20.026908 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:20.364818 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:20.364723 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" event={"ID":"d79321de-091c-47e7-8320-3471ef8f554b","Type":"ContainerStarted","Data":"409b7a0d2a7e09c9d744649318afc46121ab514d766b2e59632990c2d94d1b00"} Oct 30 18:37:20.419780 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:20.419730 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" podStartSLOduration=2.973836498 podStartE2EDuration="3.41971694s" totalImagesPullingTime="445.880442ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:20.415009355 +0000 UTC m=+2701.130579230" watchObservedRunningTime="2026-10-30 18:37:20.41971694 +0000 UTC m=+2701.135286813" Oct 30 18:37:20.746616 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:20.746538 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 18:37:21.044216 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:21.044191 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 18:37:21.373743 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:21.373654 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" Oct 30 18:37:21.773937 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:21.773896 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:21.932871 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:37:21.932836 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:37:23.461092 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:23.461001 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:25.001316 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:25.001286 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn"] Oct 30 18:37:25.007542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:25.007491 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" Oct 30 18:37:25.036710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:25.036678 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-a1a31f-serving-cert\"" Oct 30 18:37:25.041839 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:25.041813 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn"] Oct 30 18:37:25.058935 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:25.058910 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-openshift-service-ca\") pod \"maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn\" (UID: \"cb49f732-be41-4ed9-b7e6-a4eb39e146e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" Oct 30 18:37:25.059084 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:25.059013 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-metrics-certs\") pod \"maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn\" (UID: \"cb49f732-be41-4ed9-b7e6-a4eb39e146e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" Oct 30 18:37:25.059137 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:25.059107 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-maas-api-tls\") pod \"maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn\" (UID: \"cb49f732-be41-4ed9-b7e6-a4eb39e146e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" Oct 30 18:37:25.059175 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:25.059154 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vplnb\" (UniqueName: \"kubernetes.io/projected/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-kube-api-access-vplnb\") pod \"maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn\" (UID: \"cb49f732-be41-4ed9-b7e6-a4eb39e146e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" Oct 30 18:37:25.136372 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:25.136343 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:25.161068 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:25.161040 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-openshift-service-ca\") pod \"maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn\" (UID: \"cb49f732-be41-4ed9-b7e6-a4eb39e146e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" Oct 30 18:37:25.161179 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:25.161089 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-metrics-certs\") pod \"maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn\" (UID: \"cb49f732-be41-4ed9-b7e6-a4eb39e146e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" Oct 30 18:37:25.161179 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:25.161126 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-maas-api-tls\") pod \"maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn\" (UID: \"cb49f732-be41-4ed9-b7e6-a4eb39e146e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" Oct 30 18:37:25.161179 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:25.161146 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vplnb\" (UniqueName: \"kubernetes.io/projected/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-kube-api-access-vplnb\") pod \"maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn\" (UID: \"cb49f732-be41-4ed9-b7e6-a4eb39e146e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" Oct 30 18:37:25.161853 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:25.161821 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-openshift-service-ca\") pod \"maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn\" (UID: \"cb49f732-be41-4ed9-b7e6-a4eb39e146e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" Oct 30 18:37:25.163932 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:25.163906 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-maas-api-tls\") pod \"maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn\" (UID: \"cb49f732-be41-4ed9-b7e6-a4eb39e146e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" Oct 30 18:37:25.164243 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:25.164226 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-metrics-certs\") pod \"maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn\" (UID: \"cb49f732-be41-4ed9-b7e6-a4eb39e146e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" Oct 30 18:37:25.186243 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:25.186220 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vplnb\" (UniqueName: \"kubernetes.io/projected/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-kube-api-access-vplnb\") pod \"maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn\" (UID: \"cb49f732-be41-4ed9-b7e6-a4eb39e146e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" Oct 30 18:37:25.325187 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:25.324822 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" Oct 30 18:37:25.715795 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:25.715770 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn"] Oct 30 18:37:25.716658 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:37:25.716623 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb49f732_be41_4ed9_b7e6_a4eb39e146e2.slice/crio-e060def9e347242dc2913da474615dd78715d578a2a1040e92c5057507a55245 WatchSource:0}: Error finding container e060def9e347242dc2913da474615dd78715d578a2a1040e92c5057507a55245: Status 404 returned error can't find the container with id e060def9e347242dc2913da474615dd78715d578a2a1040e92c5057507a55245 Oct 30 18:37:26.462918 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:26.462838 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p"] Oct 30 18:37:26.463928 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:26.463676 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" podUID="a5a9b344-c8bb-433b-a6a2-e62c357f4101" containerName="istio-proxy" containerID="cri-o://3f3e259ff991772ad99c365783dc5153ea191f933d17f18d3da276bbd70171de" gracePeriod=30 Oct 30 18:37:26.482907 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:26.482882 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" event={"ID":"cb49f732-be41-4ed9-b7e6-a4eb39e146e2","Type":"ContainerStarted","Data":"248b7da02d2d28841419f3fabadca21412e503c09e1f391aa5491fd86823f0c3"} Oct 30 18:37:26.483072 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:26.483060 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" event={"ID":"cb49f732-be41-4ed9-b7e6-a4eb39e146e2","Type":"ContainerStarted","Data":"e060def9e347242dc2913da474615dd78715d578a2a1040e92c5057507a55245"} Oct 30 18:37:26.483279 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:26.483267 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" Oct 30 18:37:26.498310 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:26.498284 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:26.791464 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:26.791393 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" podStartSLOduration=2.482705524 podStartE2EDuration="2.791368145s" totalImagesPullingTime="308.662621ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:26.647581385 +0000 UTC m=+2707.363151260" watchObservedRunningTime="2026-10-30 18:37:26.791368145 +0000 UTC m=+2707.506938020" Oct 30 18:37:27.015745 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:27.015714 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:27.028013 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:27.027987 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:27.098872 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:27.098783 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:27.259034 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:27.259002 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:27.386428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:27.386358 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" Oct 30 18:37:27.504314 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:27.504284 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:37:28.971176 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:28.971140 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:29.825251 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:29.825217 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:30.051978 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:30.051925 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:30.194773 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:30.194651 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:30.538174 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:30.538144 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:31.769150 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.769123 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:37:31.849626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.847855 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-podinfo\") pod \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " Oct 30 18:37:31.849626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.847917 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-workload-certs\") pod \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " Oct 30 18:37:31.849626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.848067 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-token\" (UniqueName: \"kubernetes.io/projected/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-token\") pod \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " Oct 30 18:37:31.849626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.848186 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istiod-ca-cert\") pod \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " Oct 30 18:37:31.849626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.848242 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-data\") pod \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " Oct 30 18:37:31.849626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.848262 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-workload-certs" pod "a5a9b344-c8bb-433b-a6a2-e62c357f4101" (UID: "a5a9b344-c8bb-433b-a6a2-e62c357f4101"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:31.849626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.848277 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a9b344-c8bb-433b-a6a2-e62c357f4101-kube-api-access-gtmtm\" (UniqueName: \"kubernetes.io/projected/a5a9b344-c8bb-433b-a6a2-e62c357f4101-kube-api-access-gtmtm\") pod \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " Oct 30 18:37:31.849626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.848315 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-envoy\") pod \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " Oct 30 18:37:31.849626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.848380 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-workload-socket\") pod \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " Oct 30 18:37:31.849626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.848474 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-credential-socket\") pod \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\" (UID: \"a5a9b344-c8bb-433b-a6a2-e62c357f4101\") " Oct 30 18:37:31.849626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.848529 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-data" pod "a5a9b344-c8bb-433b-a6a2-e62c357f4101" (UID: "a5a9b344-c8bb-433b-a6a2-e62c357f4101"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:31.849626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.848615 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istiod-ca-cert" pod "a5a9b344-c8bb-433b-a6a2-e62c357f4101" (UID: "a5a9b344-c8bb-433b-a6a2-e62c357f4101"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:31.849626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.848903 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-workload-socket" pod "a5a9b344-c8bb-433b-a6a2-e62c357f4101" (UID: "a5a9b344-c8bb-433b-a6a2-e62c357f4101"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:31.849626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.849135 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-credential-socket" pod "a5a9b344-c8bb-433b-a6a2-e62c357f4101" (UID: "a5a9b344-c8bb-433b-a6a2-e62c357f4101"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:31.849626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.849533 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:31.849626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.849558 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:31.849626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.849573 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:31.849626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.849587 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:31.849626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.849601 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:31.851233 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.851210 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-podinfo" pod "a5a9b344-c8bb-433b-a6a2-e62c357f4101" (UID: "a5a9b344-c8bb-433b-a6a2-e62c357f4101"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:31.851349 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.851248 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a9b344-c8bb-433b-a6a2-e62c357f4101-kube-api-access-gtmtm" pod "a5a9b344-c8bb-433b-a6a2-e62c357f4101" (UID: "a5a9b344-c8bb-433b-a6a2-e62c357f4101"). InnerVolumeSpecName "kube-api-access-gtmtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:31.851446 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.851429 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-token" pod "a5a9b344-c8bb-433b-a6a2-e62c357f4101" (UID: "a5a9b344-c8bb-433b-a6a2-e62c357f4101"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:31.852992 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.852971 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-envoy" pod "a5a9b344-c8bb-433b-a6a2-e62c357f4101" (UID: "a5a9b344-c8bb-433b-a6a2-e62c357f4101"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:31.951534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.951482 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:31.951534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.951533 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:31.951702 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.951549 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtmtm\" (UniqueName: \"kubernetes.io/projected/a5a9b344-c8bb-433b-a6a2-e62c357f4101-kube-api-access-gtmtm\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:31.951702 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.951564 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a9b344-c8bb-433b-a6a2-e62c357f4101-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:31.983031 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:31.983004 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8"] Oct 30 18:37:32.154135 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:32.154101 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:32.336196 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:32.336166 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:32.498399 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:32.498324 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" Oct 30 18:37:32.575425 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:32.575395 4603 generic.go:356] "Generic (PLEG): container finished" podID="a5a9b344-c8bb-433b-a6a2-e62c357f4101" containerID="3f3e259ff991772ad99c365783dc5153ea191f933d17f18d3da276bbd70171de" exitCode=0 Oct 30 18:37:32.575627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:32.575443 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" Oct 30 18:37:32.575627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:32.575447 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" event={"ID":"a5a9b344-c8bb-433b-a6a2-e62c357f4101","Type":"ContainerDied","Data":"3f3e259ff991772ad99c365783dc5153ea191f933d17f18d3da276bbd70171de"} Oct 30 18:37:32.575627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:32.575573 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p" event={"ID":"a5a9b344-c8bb-433b-a6a2-e62c357f4101","Type":"ContainerDied","Data":"8ae379dbf029551bf3bc963e83f05143db05fd8d8d56ab23e74c7f5c8bcc1ca2"} Oct 30 18:37:32.575627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:32.575598 4603 scope.go:122] "RemoveContainer" containerID="3f3e259ff991772ad99c365783dc5153ea191f933d17f18d3da276bbd70171de" Oct 30 18:37:32.588725 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:32.588705 4603 scope.go:122] "RemoveContainer" containerID="3f3e259ff991772ad99c365783dc5153ea191f933d17f18d3da276bbd70171de" Oct 30 18:37:32.589104 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:37:32.589072 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f3e259ff991772ad99c365783dc5153ea191f933d17f18d3da276bbd70171de\": container with ID starting with 3f3e259ff991772ad99c365783dc5153ea191f933d17f18d3da276bbd70171de not found: ID does not exist" containerID="3f3e259ff991772ad99c365783dc5153ea191f933d17f18d3da276bbd70171de" Oct 30 18:37:32.589231 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:32.589100 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3e259ff991772ad99c365783dc5153ea191f933d17f18d3da276bbd70171de"} err="failed to get container status \"3f3e259ff991772ad99c365783dc5153ea191f933d17f18d3da276bbd70171de\": rpc error: code = NotFound desc = could not find container \"3f3e259ff991772ad99c365783dc5153ea191f933d17f18d3da276bbd70171de\": container with ID starting with 3f3e259ff991772ad99c365783dc5153ea191f933d17f18d3da276bbd70171de not found: ID does not exist" Oct 30 18:37:32.661807 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:32.661775 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p"] Oct 30 18:37:32.665777 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:32.665753 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-8lr9p"] Oct 30 18:37:33.411125 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:33.411093 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5a9b344-c8bb-433b-a6a2-e62c357f4101" path="/var/lib/kubelet/pods/a5a9b344-c8bb-433b-a6a2-e62c357f4101/volumes" Oct 30 18:37:34.245237 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:34.245201 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:36.028529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:36.025244 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:37.861659 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:37.860668 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt"] Oct 30 18:37:37.910188 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:37.910150 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp"] Oct 30 18:37:37.917266 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:37.917233 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:38.110903 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:38.110875 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:38.279773 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:38.279729 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:38.575105 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:38.575029 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:38.775127 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:38.775093 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:39.794533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:39.792673 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:41.529565 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:41.526518 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:42.861961 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:42.861916 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt" podUID="c7be351b-62cc-4555-84fa-a0f018a99059" containerName="payload-processing" containerID="cri-o://3a635d0121df94235e9d90badb1b899ad0a782b669c9908bf48aa8840d77c213" gracePeriod=25 Oct 30 18:37:42.911785 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:42.911721 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp" podUID="f308b20b-d5d6-41db-8376-9cb9413b13cf" containerName="payload-pre-processing" containerID="cri-o://82015ea5d91c95a00fd537a325bb2e8bd201362ee53b0e30fbce799555e309eb" gracePeriod=25 Oct 30 18:37:43.168574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:43.167889 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:44.990874 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:44.990839 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:45.072437 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:45.072399 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp" podUID="f308b20b-d5d6-41db-8376-9cb9413b13cf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:37:45.072621 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:45.072439 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt" podUID="c7be351b-62cc-4555-84fa-a0f018a99059" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:37:46.840909 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:46.840879 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:47.005740 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.005706 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" containerID="cri-o://e52baa7cb72887c1a9f1cf8e43d9bdf085371d91e39f1ea9f314ef3884e8a2f8" gracePeriod=45 Oct 30 18:37:47.282174 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.282150 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:37:47.346286 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.346259 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcebcb0f-0443-4e43-9850-e4bb46d279f4-kube-api-access-tzswl\" (UniqueName: \"kubernetes.io/projected/dcebcb0f-0443-4e43-9850-e4bb46d279f4-kube-api-access-tzswl\") pod \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " Oct 30 18:37:47.346549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.346356 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-home\" (UniqueName: \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-home\") pod \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " Oct 30 18:37:47.346549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.346427 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-tmp-dir\") pod \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " Oct 30 18:37:47.346685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.346614 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-home" pod "dcebcb0f-0443-4e43-9850-e4bb46d279f4" (UID: "dcebcb0f-0443-4e43-9850-e4bb46d279f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:47.346745 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.346703 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-model-cache\") pod \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " Oct 30 18:37:47.346810 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.346768 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-dshm\") pod \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " Oct 30 18:37:47.346810 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.346800 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcebcb0f-0443-4e43-9850-e4bb46d279f4-tls-certs\" (UniqueName: \"kubernetes.io/secret/dcebcb0f-0443-4e43-9850-e4bb46d279f4-tls-certs\") pod \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\" (UID: \"dcebcb0f-0443-4e43-9850-e4bb46d279f4\") " Oct 30 18:37:47.346929 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.346880 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-tmp-dir" pod "dcebcb0f-0443-4e43-9850-e4bb46d279f4" (UID: "dcebcb0f-0443-4e43-9850-e4bb46d279f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:47.346929 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.346916 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-model-cache" pod "dcebcb0f-0443-4e43-9850-e4bb46d279f4" (UID: "dcebcb0f-0443-4e43-9850-e4bb46d279f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:47.347591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.347565 4603 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-model-cache\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:47.347719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.347594 4603 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-home\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:47.347719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.347619 4603 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-tmp-dir\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:47.349305 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.349277 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcebcb0f-0443-4e43-9850-e4bb46d279f4-tls-certs" pod "dcebcb0f-0443-4e43-9850-e4bb46d279f4" (UID: "dcebcb0f-0443-4e43-9850-e4bb46d279f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:47.349405 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.349354 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-dshm" pod "dcebcb0f-0443-4e43-9850-e4bb46d279f4" (UID: "dcebcb0f-0443-4e43-9850-e4bb46d279f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:47.349405 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.349362 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcebcb0f-0443-4e43-9850-e4bb46d279f4-kube-api-access-tzswl" pod "dcebcb0f-0443-4e43-9850-e4bb46d279f4" (UID: "dcebcb0f-0443-4e43-9850-e4bb46d279f4"). InnerVolumeSpecName "kube-api-access-tzswl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:47.449522 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.449475 4603 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcebcb0f-0443-4e43-9850-e4bb46d279f4-dshm\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:47.449668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.449526 4603 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcebcb0f-0443-4e43-9850-e4bb46d279f4-tls-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:47.449668 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.449544 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzswl\" (UniqueName: \"kubernetes.io/projected/dcebcb0f-0443-4e43-9850-e4bb46d279f4-kube-api-access-tzswl\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:47.714073 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.713967 4603 generic.go:356] "Generic (PLEG): container finished" podID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerID="e52baa7cb72887c1a9f1cf8e43d9bdf085371d91e39f1ea9f314ef3884e8a2f8" exitCode=0 Oct 30 18:37:47.714073 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.714012 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" Oct 30 18:37:47.714073 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.714011 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" event={"ID":"dcebcb0f-0443-4e43-9850-e4bb46d279f4","Type":"ContainerDied","Data":"e52baa7cb72887c1a9f1cf8e43d9bdf085371d91e39f1ea9f314ef3884e8a2f8"} Oct 30 18:37:47.714362 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.714076 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" event={"ID":"dcebcb0f-0443-4e43-9850-e4bb46d279f4","Type":"ContainerDied","Data":"0b59fadd4b5571000b30d091411edb1a64b173ac7c0de8ed16e224a42f83a590"} Oct 30 18:37:47.714362 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.714099 4603 scope.go:122] "RemoveContainer" containerID="e52baa7cb72887c1a9f1cf8e43d9bdf085371d91e39f1ea9f314ef3884e8a2f8" Oct 30 18:37:47.724543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.724479 4603 scope.go:122] "RemoveContainer" containerID="e52baa7cb72887c1a9f1cf8e43d9bdf085371d91e39f1ea9f314ef3884e8a2f8" Oct 30 18:37:47.724825 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:37:47.724805 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e52baa7cb72887c1a9f1cf8e43d9bdf085371d91e39f1ea9f314ef3884e8a2f8\": container with ID starting with e52baa7cb72887c1a9f1cf8e43d9bdf085371d91e39f1ea9f314ef3884e8a2f8 not found: ID does not exist" containerID="e52baa7cb72887c1a9f1cf8e43d9bdf085371d91e39f1ea9f314ef3884e8a2f8" Oct 30 18:37:47.724907 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.724836 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e52baa7cb72887c1a9f1cf8e43d9bdf085371d91e39f1ea9f314ef3884e8a2f8"} err="failed to get container status \"e52baa7cb72887c1a9f1cf8e43d9bdf085371d91e39f1ea9f314ef3884e8a2f8\": rpc error: code = NotFound desc = could not find container \"e52baa7cb72887c1a9f1cf8e43d9bdf085371d91e39f1ea9f314ef3884e8a2f8\": container with ID starting with e52baa7cb72887c1a9f1cf8e43d9bdf085371d91e39f1ea9f314ef3884e8a2f8 not found: ID does not exist" Oct 30 18:37:47.777988 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.777956 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8"] Oct 30 18:37:47.788958 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:47.788940 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8"] Oct 30 18:37:48.245165 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:48.245121 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/premium-w0-57ee15-kserve-7c85cdb46-h95c8" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:37:48.287528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:48.287479 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:48.977650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:48.977622 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf"] Oct 30 18:37:48.977884 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:48.977862 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" podUID="0815ac39-de71-4bc3-98d3-28c88a9d2139" containerName="istio-proxy" containerID="cri-o://5392e5dfd5c8f6e03d4c032efc9f20da836316906e9e17868e71835c31ef1515" gracePeriod=30 Oct 30 18:37:48.989646 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:48.989620 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6"] Oct 30 18:37:48.990084 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:48.990049 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" podUID="0a8b0573-4657-4fc6-b7fd-43d8e882062d" containerName="istio-proxy" containerID="cri-o://d9e69bc11a19f762f294b24444ef876756df3428736b377eefde88c1f7b152bf" gracePeriod=30 Oct 30 18:37:49.007216 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:49.007192 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd"] Oct 30 18:37:49.007411 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:49.007384 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" podUID="d2888d82-f0fe-49af-a23d-c97b4f30be70" containerName="istio-proxy" containerID="cri-o://88c2c26dd336347a45f793eda01f76484b89023ff05551b64239f67b904da295" gracePeriod=30 Oct 30 18:37:49.126526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:49.122257 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:49.394032 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:49.393998 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:49.422750 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:49.422712 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" path="/var/lib/kubelet/pods/dcebcb0f-0443-4e43-9850-e4bb46d279f4/volumes" Oct 30 18:37:49.704575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:49.704465 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:50.337836 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:50.337795 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:37:50.338034 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:50.337863 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" podUID="d2888d82-f0fe-49af-a23d-c97b4f30be70" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:37:50.942407 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:50.942376 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:51.226132 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:51.226038 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:37:51.226652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:51.226515 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" podUID="0815ac39-de71-4bc3-98d3-28c88a9d2139" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:37:51.618271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:51.618232 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:51.669976 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:51.669943 4603 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:37:51.670321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:51.670280 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" podUID="0a8b0573-4657-4fc6-b7fd-43d8e882062d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:37:51.931022 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:37:51.930931 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:37:53.609769 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:53.606459 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:54.292497 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.292472 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:37:54.316717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.316696 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:37:54.337148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.336485 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-envoy\") pod \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " Oct 30 18:37:54.337148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.336551 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-data\") pod \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " Oct 30 18:37:54.337148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.336615 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-envoy\") pod \"0815ac39-de71-4bc3-98d3-28c88a9d2139\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " Oct 30 18:37:54.337148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.336662 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-workload-socket\") pod \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " Oct 30 18:37:54.337148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.336692 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-data\") pod \"0815ac39-de71-4bc3-98d3-28c88a9d2139\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " Oct 30 18:37:54.337148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.336721 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-workload-certs\") pod \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " Oct 30 18:37:54.337148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.336801 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-podinfo\") pod \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " Oct 30 18:37:54.337148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.336841 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-workload-socket\") pod \"0815ac39-de71-4bc3-98d3-28c88a9d2139\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " Oct 30 18:37:54.337148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.336868 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0815ac39-de71-4bc3-98d3-28c88a9d2139-kube-api-access-4zmvg\" (UniqueName: \"kubernetes.io/projected/0815ac39-de71-4bc3-98d3-28c88a9d2139-kube-api-access-4zmvg\") pod \"0815ac39-de71-4bc3-98d3-28c88a9d2139\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " Oct 30 18:37:54.337148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.336891 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-podinfo\") pod \"0815ac39-de71-4bc3-98d3-28c88a9d2139\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " Oct 30 18:37:54.337148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.336949 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a8b0573-4657-4fc6-b7fd-43d8e882062d-kube-api-access-r6p6w\" (UniqueName: \"kubernetes.io/projected/0a8b0573-4657-4fc6-b7fd-43d8e882062d-kube-api-access-r6p6w\") pod \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " Oct 30 18:37:54.337148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.336967 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-credential-socket\") pod \"0815ac39-de71-4bc3-98d3-28c88a9d2139\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " Oct 30 18:37:54.337148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.336989 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-token\" (UniqueName: \"kubernetes.io/projected/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-token\") pod \"0815ac39-de71-4bc3-98d3-28c88a9d2139\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " Oct 30 18:37:54.337148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.337041 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0815ac39-de71-4bc3-98d3-28c88a9d2139-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0815ac39-de71-4bc3-98d3-28c88a9d2139-istiod-ca-cert\") pod \"0815ac39-de71-4bc3-98d3-28c88a9d2139\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " Oct 30 18:37:54.337148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.337060 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-token\" (UniqueName: \"kubernetes.io/projected/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-token\") pod \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " Oct 30 18:37:54.339832 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.337127 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-credential-socket\") pod \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " Oct 30 18:37:54.340047 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.339949 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-workload-certs\") pod \"0815ac39-de71-4bc3-98d3-28c88a9d2139\" (UID: \"0815ac39-de71-4bc3-98d3-28c88a9d2139\") " Oct 30 18:37:54.340741 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.340158 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istiod-ca-cert\") pod \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\" (UID: \"0a8b0573-4657-4fc6-b7fd-43d8e882062d\") " Oct 30 18:37:54.341845 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.341682 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istiod-ca-cert" pod "0a8b0573-4657-4fc6-b7fd-43d8e882062d" (UID: "0a8b0573-4657-4fc6-b7fd-43d8e882062d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:54.343023 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.342997 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-data" pod "0815ac39-de71-4bc3-98d3-28c88a9d2139" (UID: "0815ac39-de71-4bc3-98d3-28c88a9d2139"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.343290 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.343267 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-data" pod "0a8b0573-4657-4fc6-b7fd-43d8e882062d" (UID: "0a8b0573-4657-4fc6-b7fd-43d8e882062d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.343759 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.343739 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.343889 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.343877 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.343987 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.343975 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.344066 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.343766 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-workload-certs" pod "0a8b0573-4657-4fc6-b7fd-43d8e882062d" (UID: "0a8b0573-4657-4fc6-b7fd-43d8e882062d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.346764 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.344956 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-envoy" pod "0a8b0573-4657-4fc6-b7fd-43d8e882062d" (UID: "0a8b0573-4657-4fc6-b7fd-43d8e882062d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.346764 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.345488 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-credential-socket" pod "0a8b0573-4657-4fc6-b7fd-43d8e882062d" (UID: "0a8b0573-4657-4fc6-b7fd-43d8e882062d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.346764 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.345744 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-workload-certs" pod "0815ac39-de71-4bc3-98d3-28c88a9d2139" (UID: "0815ac39-de71-4bc3-98d3-28c88a9d2139"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.346764 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.345950 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-workload-socket" pod "0a8b0573-4657-4fc6-b7fd-43d8e882062d" (UID: "0a8b0573-4657-4fc6-b7fd-43d8e882062d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.346764 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.346188 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-workload-socket" pod "0815ac39-de71-4bc3-98d3-28c88a9d2139" (UID: "0815ac39-de71-4bc3-98d3-28c88a9d2139"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.346764 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.346369 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-credential-socket" pod "0815ac39-de71-4bc3-98d3-28c88a9d2139" (UID: "0815ac39-de71-4bc3-98d3-28c88a9d2139"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.346764 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.346724 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0815ac39-de71-4bc3-98d3-28c88a9d2139-istiod-ca-cert" pod "0815ac39-de71-4bc3-98d3-28c88a9d2139" (UID: "0815ac39-de71-4bc3-98d3-28c88a9d2139"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:54.349624 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.349149 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-token" pod "0a8b0573-4657-4fc6-b7fd-43d8e882062d" (UID: "0a8b0573-4657-4fc6-b7fd-43d8e882062d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:54.351895 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.351850 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-podinfo" pod "0815ac39-de71-4bc3-98d3-28c88a9d2139" (UID: "0815ac39-de71-4bc3-98d3-28c88a9d2139"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:54.351895 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.351842 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-podinfo" pod "0a8b0573-4657-4fc6-b7fd-43d8e882062d" (UID: "0a8b0573-4657-4fc6-b7fd-43d8e882062d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:54.358901 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.358863 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-envoy" pod "0815ac39-de71-4bc3-98d3-28c88a9d2139" (UID: "0815ac39-de71-4bc3-98d3-28c88a9d2139"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.359027 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.358927 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-token" pod "0815ac39-de71-4bc3-98d3-28c88a9d2139" (UID: "0815ac39-de71-4bc3-98d3-28c88a9d2139"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:54.359090 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.359047 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0815ac39-de71-4bc3-98d3-28c88a9d2139-kube-api-access-4zmvg" pod "0815ac39-de71-4bc3-98d3-28c88a9d2139" (UID: "0815ac39-de71-4bc3-98d3-28c88a9d2139"). InnerVolumeSpecName "kube-api-access-4zmvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:54.361121 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.361095 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8b0573-4657-4fc6-b7fd-43d8e882062d-kube-api-access-r6p6w" pod "0a8b0573-4657-4fc6-b7fd-43d8e882062d" (UID: "0a8b0573-4657-4fc6-b7fd-43d8e882062d"). InnerVolumeSpecName "kube-api-access-r6p6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:54.384242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.384214 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:37:54.445035 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.444995 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-workload-certs\") pod \"d2888d82-f0fe-49af-a23d-c97b4f30be70\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " Oct 30 18:37:54.445199 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.445045 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-workload-socket\") pod \"d2888d82-f0fe-49af-a23d-c97b4f30be70\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " Oct 30 18:37:54.445199 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.445119 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2888d82-f0fe-49af-a23d-c97b4f30be70-kube-api-access-n9r5p\" (UniqueName: \"kubernetes.io/projected/d2888d82-f0fe-49af-a23d-c97b4f30be70-kube-api-access-n9r5p\") pod \"d2888d82-f0fe-49af-a23d-c97b4f30be70\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " Oct 30 18:37:54.445199 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.445188 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2888d82-f0fe-49af-a23d-c97b4f30be70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2888d82-f0fe-49af-a23d-c97b4f30be70-istiod-ca-cert\") pod \"d2888d82-f0fe-49af-a23d-c97b4f30be70\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " Oct 30 18:37:54.445394 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.445227 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-envoy\") pod \"d2888d82-f0fe-49af-a23d-c97b4f30be70\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " Oct 30 18:37:54.445394 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.445333 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-workload-certs" pod "d2888d82-f0fe-49af-a23d-c97b4f30be70" (UID: "d2888d82-f0fe-49af-a23d-c97b4f30be70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.445394 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.445379 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-podinfo\") pod \"d2888d82-f0fe-49af-a23d-c97b4f30be70\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " Oct 30 18:37:54.445646 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.445430 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-credential-socket\") pod \"d2888d82-f0fe-49af-a23d-c97b4f30be70\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " Oct 30 18:37:54.447562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.445639 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2888d82-f0fe-49af-a23d-c97b4f30be70-istiod-ca-cert" pod "d2888d82-f0fe-49af-a23d-c97b4f30be70" (UID: "d2888d82-f0fe-49af-a23d-c97b4f30be70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:54.447562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.445723 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-credential-socket" pod "d2888d82-f0fe-49af-a23d-c97b4f30be70" (UID: "d2888d82-f0fe-49af-a23d-c97b4f30be70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.447562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.445804 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-data\") pod \"d2888d82-f0fe-49af-a23d-c97b4f30be70\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " Oct 30 18:37:54.447562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.445891 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-token\" (UniqueName: \"kubernetes.io/projected/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-token\") pod \"d2888d82-f0fe-49af-a23d-c97b4f30be70\" (UID: \"d2888d82-f0fe-49af-a23d-c97b4f30be70\") " Oct 30 18:37:54.447562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.446121 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-data" pod "d2888d82-f0fe-49af-a23d-c97b4f30be70" (UID: "d2888d82-f0fe-49af-a23d-c97b4f30be70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.447562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.446218 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-workload-socket" pod "d2888d82-f0fe-49af-a23d-c97b4f30be70" (UID: "d2888d82-f0fe-49af-a23d-c97b4f30be70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.447965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.447888 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.447965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.447910 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.447965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.447926 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.447965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.447939 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.447965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.447954 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.448239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.447967 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.448239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.447979 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.448239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.447996 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.448239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.448010 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zmvg\" (UniqueName: \"kubernetes.io/projected/0815ac39-de71-4bc3-98d3-28c88a9d2139-kube-api-access-4zmvg\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.448239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.448024 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.448239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.448037 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6p6w\" (UniqueName: \"kubernetes.io/projected/0a8b0573-4657-4fc6-b7fd-43d8e882062d-kube-api-access-r6p6w\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.448239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.448051 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.448239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.448064 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0815ac39-de71-4bc3-98d3-28c88a9d2139-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.448239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.448079 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.448239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.448092 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.448239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.448105 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0815ac39-de71-4bc3-98d3-28c88a9d2139-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.448239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.448118 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8b0573-4657-4fc6-b7fd-43d8e882062d-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.448239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.448131 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8b0573-4657-4fc6-b7fd-43d8e882062d-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.448239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.448145 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0815ac39-de71-4bc3-98d3-28c88a9d2139-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.448239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.448159 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2888d82-f0fe-49af-a23d-c97b4f30be70-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.448884 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.448663 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-envoy" pod "d2888d82-f0fe-49af-a23d-c97b4f30be70" (UID: "d2888d82-f0fe-49af-a23d-c97b4f30be70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.448884 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.448760 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-podinfo" pod "d2888d82-f0fe-49af-a23d-c97b4f30be70" (UID: "d2888d82-f0fe-49af-a23d-c97b4f30be70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:54.448956 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.448918 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2888d82-f0fe-49af-a23d-c97b4f30be70-kube-api-access-n9r5p" pod "d2888d82-f0fe-49af-a23d-c97b4f30be70" (UID: "d2888d82-f0fe-49af-a23d-c97b4f30be70"). InnerVolumeSpecName "kube-api-access-n9r5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:54.450093 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.450069 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-token" pod "d2888d82-f0fe-49af-a23d-c97b4f30be70" (UID: "d2888d82-f0fe-49af-a23d-c97b4f30be70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:54.549666 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.549634 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.549666 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.549661 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9r5p\" (UniqueName: \"kubernetes.io/projected/d2888d82-f0fe-49af-a23d-c97b4f30be70-kube-api-access-n9r5p\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.549666 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.549672 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.549904 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.549681 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2888d82-f0fe-49af-a23d-c97b4f30be70-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.633938 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.633899 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:54.766801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.766772 4603 generic.go:356] "Generic (PLEG): container finished" podID="0a8b0573-4657-4fc6-b7fd-43d8e882062d" containerID="d9e69bc11a19f762f294b24444ef876756df3428736b377eefde88c1f7b152bf" exitCode=0 Oct 30 18:37:54.766990 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.766838 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" Oct 30 18:37:54.766990 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.766859 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" event={"ID":"0a8b0573-4657-4fc6-b7fd-43d8e882062d","Type":"ContainerDied","Data":"d9e69bc11a19f762f294b24444ef876756df3428736b377eefde88c1f7b152bf"} Oct 30 18:37:54.766990 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.766916 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6" event={"ID":"0a8b0573-4657-4fc6-b7fd-43d8e882062d","Type":"ContainerDied","Data":"48c01fcbd28167ba71e40c89aa7a7585b6100333030532d2c02b9cfdf63feb7d"} Oct 30 18:37:54.766990 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.766946 4603 scope.go:122] "RemoveContainer" containerID="d9e69bc11a19f762f294b24444ef876756df3428736b377eefde88c1f7b152bf" Oct 30 18:37:54.768789 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.768759 4603 generic.go:356] "Generic (PLEG): container finished" podID="d2888d82-f0fe-49af-a23d-c97b4f30be70" containerID="88c2c26dd336347a45f793eda01f76484b89023ff05551b64239f67b904da295" exitCode=0 Oct 30 18:37:54.768916 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.768869 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" event={"ID":"d2888d82-f0fe-49af-a23d-c97b4f30be70","Type":"ContainerDied","Data":"88c2c26dd336347a45f793eda01f76484b89023ff05551b64239f67b904da295"} Oct 30 18:37:54.768964 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.768914 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" event={"ID":"d2888d82-f0fe-49af-a23d-c97b4f30be70","Type":"ContainerDied","Data":"2f083e0c5f2d5823c20eb23d2999fbb8819e0821632e480a83498dad2c93c0fa"} Oct 30 18:37:54.769007 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.768986 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd" Oct 30 18:37:54.779179 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.779157 4603 generic.go:356] "Generic (PLEG): container finished" podID="0815ac39-de71-4bc3-98d3-28c88a9d2139" containerID="5392e5dfd5c8f6e03d4c032efc9f20da836316906e9e17868e71835c31ef1515" exitCode=0 Oct 30 18:37:54.779328 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.779303 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" Oct 30 18:37:54.779463 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.779325 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" event={"ID":"0815ac39-de71-4bc3-98d3-28c88a9d2139","Type":"ContainerDied","Data":"5392e5dfd5c8f6e03d4c032efc9f20da836316906e9e17868e71835c31ef1515"} Oct 30 18:37:54.779463 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.779382 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf" event={"ID":"0815ac39-de71-4bc3-98d3-28c88a9d2139","Type":"ContainerDied","Data":"a117584a514a6056c2485be52a8a8e63088b3d5fb6389df917dd23a741b82d72"} Oct 30 18:37:54.782808 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.782793 4603 scope.go:122] "RemoveContainer" containerID="d9e69bc11a19f762f294b24444ef876756df3428736b377eefde88c1f7b152bf" Oct 30 18:37:54.783065 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:37:54.783045 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9e69bc11a19f762f294b24444ef876756df3428736b377eefde88c1f7b152bf\": container with ID starting with d9e69bc11a19f762f294b24444ef876756df3428736b377eefde88c1f7b152bf not found: ID does not exist" containerID="d9e69bc11a19f762f294b24444ef876756df3428736b377eefde88c1f7b152bf" Oct 30 18:37:54.783157 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.783075 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9e69bc11a19f762f294b24444ef876756df3428736b377eefde88c1f7b152bf"} err="failed to get container status \"d9e69bc11a19f762f294b24444ef876756df3428736b377eefde88c1f7b152bf\": rpc error: code = NotFound desc = could not find container \"d9e69bc11a19f762f294b24444ef876756df3428736b377eefde88c1f7b152bf\": container with ID starting with d9e69bc11a19f762f294b24444ef876756df3428736b377eefde88c1f7b152bf not found: ID does not exist" Oct 30 18:37:54.783157 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.783112 4603 scope.go:122] "RemoveContainer" containerID="88c2c26dd336347a45f793eda01f76484b89023ff05551b64239f67b904da295" Oct 30 18:37:54.794242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.794220 4603 scope.go:122] "RemoveContainer" containerID="88c2c26dd336347a45f793eda01f76484b89023ff05551b64239f67b904da295" Oct 30 18:37:54.794535 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:37:54.794480 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88c2c26dd336347a45f793eda01f76484b89023ff05551b64239f67b904da295\": container with ID starting with 88c2c26dd336347a45f793eda01f76484b89023ff05551b64239f67b904da295 not found: ID does not exist" containerID="88c2c26dd336347a45f793eda01f76484b89023ff05551b64239f67b904da295" Oct 30 18:37:54.794631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.794585 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c2c26dd336347a45f793eda01f76484b89023ff05551b64239f67b904da295"} err="failed to get container status \"88c2c26dd336347a45f793eda01f76484b89023ff05551b64239f67b904da295\": rpc error: code = NotFound desc = could not find container \"88c2c26dd336347a45f793eda01f76484b89023ff05551b64239f67b904da295\": container with ID starting with 88c2c26dd336347a45f793eda01f76484b89023ff05551b64239f67b904da295 not found: ID does not exist" Oct 30 18:37:54.794631 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.794611 4603 scope.go:122] "RemoveContainer" containerID="5392e5dfd5c8f6e03d4c032efc9f20da836316906e9e17868e71835c31ef1515" Oct 30 18:37:54.804941 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.804925 4603 scope.go:122] "RemoveContainer" containerID="7e367247a2f0e8c781fafc99df6353df88fab7c94e13841a9887e8c5c86603da" Oct 30 18:37:54.814190 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.814173 4603 scope.go:122] "RemoveContainer" containerID="5392e5dfd5c8f6e03d4c032efc9f20da836316906e9e17868e71835c31ef1515" Oct 30 18:37:54.814472 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:37:54.814452 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5392e5dfd5c8f6e03d4c032efc9f20da836316906e9e17868e71835c31ef1515\": container with ID starting with 5392e5dfd5c8f6e03d4c032efc9f20da836316906e9e17868e71835c31ef1515 not found: ID does not exist" containerID="5392e5dfd5c8f6e03d4c032efc9f20da836316906e9e17868e71835c31ef1515" Oct 30 18:37:54.814569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.814478 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5392e5dfd5c8f6e03d4c032efc9f20da836316906e9e17868e71835c31ef1515"} err="failed to get container status \"5392e5dfd5c8f6e03d4c032efc9f20da836316906e9e17868e71835c31ef1515\": rpc error: code = NotFound desc = could not find container \"5392e5dfd5c8f6e03d4c032efc9f20da836316906e9e17868e71835c31ef1515\": container with ID starting with 5392e5dfd5c8f6e03d4c032efc9f20da836316906e9e17868e71835c31ef1515 not found: ID does not exist" Oct 30 18:37:54.814569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.814492 4603 scope.go:122] "RemoveContainer" containerID="7e367247a2f0e8c781fafc99df6353df88fab7c94e13841a9887e8c5c86603da" Oct 30 18:37:54.814803 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:37:54.814785 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e367247a2f0e8c781fafc99df6353df88fab7c94e13841a9887e8c5c86603da\": container with ID starting with 7e367247a2f0e8c781fafc99df6353df88fab7c94e13841a9887e8c5c86603da not found: ID does not exist" containerID="7e367247a2f0e8c781fafc99df6353df88fab7c94e13841a9887e8c5c86603da" Oct 30 18:37:54.814861 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.814812 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e367247a2f0e8c781fafc99df6353df88fab7c94e13841a9887e8c5c86603da"} err="failed to get container status \"7e367247a2f0e8c781fafc99df6353df88fab7c94e13841a9887e8c5c86603da\": rpc error: code = NotFound desc = could not find container \"7e367247a2f0e8c781fafc99df6353df88fab7c94e13841a9887e8c5c86603da\": container with ID starting with 7e367247a2f0e8c781fafc99df6353df88fab7c94e13841a9887e8c5c86603da not found: ID does not exist" Oct 30 18:37:54.838584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.838563 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd"] Oct 30 18:37:54.849947 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.849927 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nr4vd"] Oct 30 18:37:54.875767 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.875743 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6"] Oct 30 18:37:54.892358 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.892329 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-7tss6"] Oct 30 18:37:54.928658 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.928634 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf"] Oct 30 18:37:54.932899 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:54.932879 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-nc5xf"] Oct 30 18:37:55.107826 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:55.107716 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp" podUID="f308b20b-d5d6-41db-8376-9cb9413b13cf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:37:55.108707 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:55.108676 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt" podUID="c7be351b-62cc-4555-84fa-a0f018a99059" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:37:55.377569 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:55.377461 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:55.431468 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:55.431401 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0815ac39-de71-4bc3-98d3-28c88a9d2139" path="/var/lib/kubelet/pods/0815ac39-de71-4bc3-98d3-28c88a9d2139/volumes" Oct 30 18:37:55.432299 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:55.432272 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a8b0573-4657-4fc6-b7fd-43d8e882062d" path="/var/lib/kubelet/pods/0a8b0573-4657-4fc6-b7fd-43d8e882062d/volumes" Oct 30 18:37:55.433240 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:55.433199 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2888d82-f0fe-49af-a23d-c97b4f30be70" path="/var/lib/kubelet/pods/d2888d82-f0fe-49af-a23d-c97b4f30be70/volumes" Oct 30 18:37:56.117026 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:56.116991 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:57.282130 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:57.282095 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:59.299627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:37:59.295088 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:00.105892 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:00.105858 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:38:00.833203 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:00.833166 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:01.019980 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:01.019948 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:01.954186 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:01.954153 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:04.631551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:04.625758 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Oct 30 18:38:04.631551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:04.626238 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" podUID="81d31061-bdab-430c-9dba-d50879b97a29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.176:8000/ready\": context deadline exceeded" Oct 30 18:38:04.664768 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:04.663922 4603 generic.go:356] "Generic (PLEG): container finished" podID="c7be351b-62cc-4555-84fa-a0f018a99059" containerID="3a635d0121df94235e9d90badb1b899ad0a782b669c9908bf48aa8840d77c213" exitCode=0 Oct 30 18:38:04.664768 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:04.664182 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt" event={"ID":"c7be351b-62cc-4555-84fa-a0f018a99059","Type":"ContainerDied","Data":"3a635d0121df94235e9d90badb1b899ad0a782b669c9908bf48aa8840d77c213"} Oct 30 18:38:04.741559 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:04.736830 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:04.741559 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:04.739442 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt" Oct 30 18:38:04.764537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:04.745662 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp" Oct 30 18:38:04.765518 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:04.765473 4603 generic.go:356] "Generic (PLEG): container finished" podID="f308b20b-d5d6-41db-8376-9cb9413b13cf" containerID="82015ea5d91c95a00fd537a325bb2e8bd201362ee53b0e30fbce799555e309eb" exitCode=0 Oct 30 18:38:04.765763 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:04.765742 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp" event={"ID":"f308b20b-d5d6-41db-8376-9cb9413b13cf","Type":"ContainerDied","Data":"82015ea5d91c95a00fd537a325bb2e8bd201362ee53b0e30fbce799555e309eb"} Oct 30 18:38:04.765918 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:04.765903 4603 scope.go:122] "RemoveContainer" containerID="82015ea5d91c95a00fd537a325bb2e8bd201362ee53b0e30fbce799555e309eb" Oct 30 18:38:04.808952 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:04.807612 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:04.883261 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:04.883221 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7be351b-62cc-4555-84fa-a0f018a99059-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7be351b-62cc-4555-84fa-a0f018a99059-plugins-config-volume\") pod \"c7be351b-62cc-4555-84fa-a0f018a99059\" (UID: \"c7be351b-62cc-4555-84fa-a0f018a99059\") " Oct 30 18:38:04.883813 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:04.883783 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7be351b-62cc-4555-84fa-a0f018a99059-kube-api-access-zg9fm\" (UniqueName: \"kubernetes.io/projected/c7be351b-62cc-4555-84fa-a0f018a99059-kube-api-access-zg9fm\") pod \"c7be351b-62cc-4555-84fa-a0f018a99059\" (UID: \"c7be351b-62cc-4555-84fa-a0f018a99059\") " Oct 30 18:38:04.884041 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:04.884024 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f308b20b-d5d6-41db-8376-9cb9413b13cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f308b20b-d5d6-41db-8376-9cb9413b13cf-plugins-config-volume\") pod \"f308b20b-d5d6-41db-8376-9cb9413b13cf\" (UID: \"f308b20b-d5d6-41db-8376-9cb9413b13cf\") " Oct 30 18:38:04.884281 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:04.884264 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f308b20b-d5d6-41db-8376-9cb9413b13cf-kube-api-access-jmwtq\" (UniqueName: \"kubernetes.io/projected/f308b20b-d5d6-41db-8376-9cb9413b13cf-kube-api-access-jmwtq\") pod \"f308b20b-d5d6-41db-8376-9cb9413b13cf\" (UID: \"f308b20b-d5d6-41db-8376-9cb9413b13cf\") " Oct 30 18:38:04.907311 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:04.907279 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f308b20b-d5d6-41db-8376-9cb9413b13cf-plugins-config-volume" pod "f308b20b-d5d6-41db-8376-9cb9413b13cf" (UID: "f308b20b-d5d6-41db-8376-9cb9413b13cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:04.908062 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:04.908034 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7be351b-62cc-4555-84fa-a0f018a99059-plugins-config-volume" pod "c7be351b-62cc-4555-84fa-a0f018a99059" (UID: "c7be351b-62cc-4555-84fa-a0f018a99059"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:05.038537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:04.991169 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f308b20b-d5d6-41db-8376-9cb9413b13cf-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:05.038537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:04.991213 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7be351b-62cc-4555-84fa-a0f018a99059-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:05.038537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:05.019655 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:05.063584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:05.052416 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f308b20b-d5d6-41db-8376-9cb9413b13cf-kube-api-access-jmwtq" pod "f308b20b-d5d6-41db-8376-9cb9413b13cf" (UID: "f308b20b-d5d6-41db-8376-9cb9413b13cf"). InnerVolumeSpecName "kube-api-access-jmwtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:05.156725 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:05.064054 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7be351b-62cc-4555-84fa-a0f018a99059-kube-api-access-zg9fm" pod "c7be351b-62cc-4555-84fa-a0f018a99059" (UID: "c7be351b-62cc-4555-84fa-a0f018a99059"). InnerVolumeSpecName "kube-api-access-zg9fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:05.156725 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:05.110697 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:05.156725 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:05.110613 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:38:05.156725 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:05.110723 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:38:05.245527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:05.240205 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg9fm\" (UniqueName: \"kubernetes.io/projected/c7be351b-62cc-4555-84fa-a0f018a99059-kube-api-access-zg9fm\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:05.245527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:05.240252 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmwtq\" (UniqueName: \"kubernetes.io/projected/f308b20b-d5d6-41db-8376-9cb9413b13cf-kube-api-access-jmwtq\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:05.362584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:05.351189 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:05.514750 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:05.514708 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:05.735595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:05.669551 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" podUID="81d31061-bdab-430c-9dba-d50879b97a29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.176:8000/ready\": context deadline exceeded" Oct 30 18:38:05.766062 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:05.737259 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:38:05.775943 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:05.766908 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:38:05.803928 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:05.803898 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:38:05.920879 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:05.919943 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp" Oct 30 18:38:05.920879 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:05.920805 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp" event={"ID":"f308b20b-d5d6-41db-8376-9cb9413b13cf","Type":"ContainerDied","Data":"731b04f8d69d055f825f9887a17c09397e23236db2d77c3a8427e43d67f048a7"} Oct 30 18:38:05.952047 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:05.952010 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt" event={"ID":"c7be351b-62cc-4555-84fa-a0f018a99059","Type":"ContainerDied","Data":"ef53ab62af9d7e8fd12a9fa0e49d6b863d10195fd32a13bcb132105a2394d975"} Oct 30 18:38:05.964645 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:05.964611 4603 scope.go:122] "RemoveContainer" containerID="3a635d0121df94235e9d90badb1b899ad0a782b669c9908bf48aa8840d77c213" Oct 30 18:38:05.964964 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:05.963556 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt" Oct 30 18:38:06.182589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:06.113133 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:38:06.223766 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:06.134022 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:38:06.377831 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:06.356238 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:06.377831 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:06.185973 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:38:06.377831 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:06.197743 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:38:06.499586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:06.179385 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:38:06.499586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:06.494761 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:38:06.499586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:06.494959 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:38:06.499586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:06.495366 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:38:06.499586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:06.495436 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:38:06.499586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:06.495827 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:38:06.781839 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:06.781789 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:07.169589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:07.077778 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" podUID="81d31061-bdab-430c-9dba-d50879b97a29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.176:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:07.229528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:07.137643 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:07.399533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:07.358291 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:38:07.399533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:07.358401 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded" Oct 30 18:38:07.399533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:07.358465 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:07.399533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:07.366816 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:38:07.501977 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:07.501868 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:07.583688 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:07.583638 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:07.938813 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:07.938768 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:38:08.148096 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:08.147997 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" podUID="81d31061-bdab-430c-9dba-d50879b97a29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.176:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:08.211330 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:08.211283 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" podUID="81d31061-bdab-430c-9dba-d50879b97a29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.176:8000/ready\": context deadline exceeded" Oct 30 18:38:08.251692 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:08.251633 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:38:08.274767 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:08.274721 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:08.285998 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:08.285963 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:38:08.295815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:08.290341 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" podUID="d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:08.368802 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:08.368761 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:38:08.370719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:08.370693 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:38:08.378643 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:08.378600 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp"] Oct 30 18:38:08.378865 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:08.378850 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-57ee15-7d67c54c5f-srwsp"] Oct 30 18:38:08.378999 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:08.378989 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt"] Oct 30 18:38:08.379110 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:08.379100 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-57ee15-7558957ff4-642nt"] Oct 30 18:38:08.448061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:08.447967 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:38:08.457997 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:08.457932 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" podUID="58e2469b-dc0d-40ab-bd74-a47c4a9e438b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:08.487589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:08.487485 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:38:08.487905 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:08.487860 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" podUID="f7882513-ef6d-4b11-871e-39a29ad022df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:08.508232 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:08.508188 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:08.590794 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:08.587091 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:08.600264 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:08.600226 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:08.659724 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:08.659665 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:08.661543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:08.659979 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:08.661751 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:08.660038 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 30 18:38:08.980529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:08.951078 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:38:09.217348 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:09.216635 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" podUID="81d31061-bdab-430c-9dba-d50879b97a29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.176:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:09.261615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:09.252583 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:09.389532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:09.388609 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:09.544201 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:09.544158 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 18:38:09.587530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:09.561364 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7be351b-62cc-4555-84fa-a0f018a99059" path="/var/lib/kubelet/pods/c7be351b-62cc-4555-84fa-a0f018a99059/volumes" Oct 30 18:38:09.609912 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:09.609874 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f308b20b-d5d6-41db-8376-9cb9413b13cf" path="/var/lib/kubelet/pods/f308b20b-d5d6-41db-8376-9cb9413b13cf/volumes" Oct 30 18:38:09.610778 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:09.608726 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:38:09.611985 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:09.589052 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:38:09.688679 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:09.688637 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:09.857141 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:09.856609 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:38:09.857141 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:09.856675 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" podUID="47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:09.984287 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:09.984244 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:38:10.223170 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:10.223068 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" podUID="81d31061-bdab-430c-9dba-d50879b97a29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.176:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:10.258722 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:10.258675 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:38:10.283695 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:10.283655 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:38:10.284025 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:10.283994 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" podUID="db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:10.392663 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:10.392614 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:38:10.545634 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:10.545587 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:10.611755 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:10.611705 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:10.612311 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:10.612284 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:10.691651 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:10.691602 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:10.999587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:10.995069 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:38:11.206527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:11.189238 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:11.229559 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:11.228061 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" podUID="81d31061-bdab-430c-9dba-d50879b97a29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.176:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:11.269807 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:11.269769 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:38:11.396636 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:11.396587 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:11.549762 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:11.549648 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:11.559724 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:11.558653 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:38:11.559724 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:11.558714 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" podUID="c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:11.616754 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:11.616661 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:38:11.618107 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:11.618073 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:11.778690 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:11.778620 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:12.005433 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:12.005320 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:12.237707 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:12.237655 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" podUID="81d31061-bdab-430c-9dba-d50879b97a29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.176:8000/ready\": context deadline exceeded" Oct 30 18:38:12.242636 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:12.242599 4603 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:38:12.242896 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:12.242866 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" podUID="d5567046-aa24-421a-8339-fdd9cf56e62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:12.300231 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:12.300185 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:12.411088 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:12.410807 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:38:12.573168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:12.573046 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:12.640810 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:12.640750 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 18:38:12.642764 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:12.642729 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:12.854579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:12.851029 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:38:13.028539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:13.026596 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:38:13.262589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:13.245143 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" podUID="81d31061-bdab-430c-9dba-d50879b97a29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.176:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:13.404528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:13.385139 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:13.566288 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:13.422873 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:13.593546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:13.426952 4603 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:38:13.593546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:13.585402 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" podUID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:13.593546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:13.556242 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:13.615549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:13.593975 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:13.666554 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:13.642948 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:13.666554 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:13.645204 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:13.854022 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:13.853543 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:14.034644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:14.033867 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:14.245711 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:14.245602 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" podUID="81d31061-bdab-430c-9dba-d50879b97a29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.176:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:14.357260 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:14.356270 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:38:14.383016 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:14.382986 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:38:14.386361 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:14.384384 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:38:14.567225 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:14.567194 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:38:14.571317 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:14.569051 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 18:38:14.579653 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:14.579630 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:38:14.579862 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:14.579850 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:38:14.586334 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:14.586318 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:38:14.621625 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:14.621589 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:38:15.221520 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:15.221463 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:15.427139 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:15.427099 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:16.017287 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:16.016460 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:18.115013 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:18.114980 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:18.349376 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:18.349336 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:18.562082 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:18.562051 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:19.233332 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:19.233296 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:19.996849 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:19.992935 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:21.878871 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:21.878822 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:21.983859 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:38:21.983818 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:38:22.087674 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:22.087630 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:38:22.087850 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:22.087699 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" podUID="b6ce7227-01e3-486c-93eb-f945903f32cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:22.766666 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:22.766641 4603 generic.go:356] "Generic (PLEG): container finished" podID="b6ce7227-01e3-486c-93eb-f945903f32cf" containerID="13f370e1c91d6b9858fa11610abd2d2b7a6e100840f20bcefa2e44363dac64be" exitCode=0 Oct 30 18:38:22.766851 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:22.766701 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" event={"ID":"b6ce7227-01e3-486c-93eb-f945903f32cf","Type":"ContainerDied","Data":"13f370e1c91d6b9858fa11610abd2d2b7a6e100840f20bcefa2e44363dac64be"} Oct 30 18:38:22.767085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:22.767071 4603 scope.go:122] "RemoveContainer" containerID="13f370e1c91d6b9858fa11610abd2d2b7a6e100840f20bcefa2e44363dac64be" Oct 30 18:38:22.768674 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:22.768644 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:38:22.768769 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:22.768715 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:38:22.768769 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:22.768749 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:38:23.775473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:23.775439 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" event={"ID":"b6ce7227-01e3-486c-93eb-f945903f32cf","Type":"ContainerStarted","Data":"a8718679c05bb9ac84c92c970d87ec8e374f27644d90aa3abe19df52bd623562"} Oct 30 18:38:23.975532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:23.971890 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:24.766650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:24.766622 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:38:24.766816 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:24.766785 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 18:38:24.766887 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:24.766832 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" podUID="b6ce7227-01e3-486c-93eb-f945903f32cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 18:38:25.766407 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:25.766374 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 18:38:25.766836 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:25.766432 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" podUID="b6ce7227-01e3-486c-93eb-f945903f32cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 18:38:26.149195 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:26.149140 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:26.419467 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:26.419388 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:26.766795 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:26.766723 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 18:38:26.766795 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:26.766782 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" podUID="b6ce7227-01e3-486c-93eb-f945903f32cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 18:38:27.766764 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:27.766729 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 18:38:27.766947 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:27.766784 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" podUID="b6ce7227-01e3-486c-93eb-f945903f32cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 18:38:27.776554 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:27.776490 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:28.235180 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:28.235134 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:28.766278 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:28.766244 4603 patch_prober.go:29] interesting pod/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 18:38:28.766460 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:28.766309 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" podUID="b6ce7227-01e3-486c-93eb-f945903f32cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 18:38:29.770778 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:29.770748 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:38:29.771173 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:29.771155 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:38:29.771934 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:29.771910 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:38:30.659741 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:30.659707 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:32.413246 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:32.413210 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:32.807084 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:32.807048 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:33.604779 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:33.604742 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:34.091244 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:34.091208 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:34.368318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:34.368222 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:35.429632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:35.429484 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2"] Oct 30 18:38:35.430529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:35.429740 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" podUID="e0c05c4b-da86-4622-88ec-960248babce5" containerName="maas-api" containerID="cri-o://0124c9526b3d7458ed3417d383b5321f48f66eb01120b4bfc1ed62f6e1cc5f04" gracePeriod=30 Oct 30 18:38:35.548369 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:35.548333 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:35.799759 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:35.799728 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" Oct 30 18:38:35.917760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:35.917718 4603 generic.go:356] "Generic (PLEG): container finished" podID="e0c05c4b-da86-4622-88ec-960248babce5" containerID="0124c9526b3d7458ed3417d383b5321f48f66eb01120b4bfc1ed62f6e1cc5f04" exitCode=0 Oct 30 18:38:35.918212 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:35.918167 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" event={"ID":"e0c05c4b-da86-4622-88ec-960248babce5","Type":"ContainerDied","Data":"0124c9526b3d7458ed3417d383b5321f48f66eb01120b4bfc1ed62f6e1cc5f04"} Oct 30 18:38:35.918432 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:35.918397 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" event={"ID":"e0c05c4b-da86-4622-88ec-960248babce5","Type":"ContainerDied","Data":"c88b02632211660a408b58a25e98da5733ed5f0ffa391b00348dca39ef48895d"} Oct 30 18:38:35.918644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:35.918606 4603 scope.go:122] "RemoveContainer" containerID="0124c9526b3d7458ed3417d383b5321f48f66eb01120b4bfc1ed62f6e1cc5f04" Oct 30 18:38:35.919011 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:35.918985 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2" Oct 30 18:38:35.937591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:35.937444 4603 scope.go:122] "RemoveContainer" containerID="0124c9526b3d7458ed3417d383b5321f48f66eb01120b4bfc1ed62f6e1cc5f04" Oct 30 18:38:35.939205 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:38:35.939132 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0124c9526b3d7458ed3417d383b5321f48f66eb01120b4bfc1ed62f6e1cc5f04\": container with ID starting with 0124c9526b3d7458ed3417d383b5321f48f66eb01120b4bfc1ed62f6e1cc5f04 not found: ID does not exist" containerID="0124c9526b3d7458ed3417d383b5321f48f66eb01120b4bfc1ed62f6e1cc5f04" Oct 30 18:38:35.939205 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:35.939165 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0124c9526b3d7458ed3417d383b5321f48f66eb01120b4bfc1ed62f6e1cc5f04"} err="failed to get container status \"0124c9526b3d7458ed3417d383b5321f48f66eb01120b4bfc1ed62f6e1cc5f04\": rpc error: code = NotFound desc = could not find container \"0124c9526b3d7458ed3417d383b5321f48f66eb01120b4bfc1ed62f6e1cc5f04\": container with ID starting with 0124c9526b3d7458ed3417d383b5321f48f66eb01120b4bfc1ed62f6e1cc5f04 not found: ID does not exist" Oct 30 18:38:35.956387 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:35.954967 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0c05c4b-da86-4622-88ec-960248babce5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0c05c4b-da86-4622-88ec-960248babce5-metrics-certs\") pod \"e0c05c4b-da86-4622-88ec-960248babce5\" (UID: \"e0c05c4b-da86-4622-88ec-960248babce5\") " Oct 30 18:38:35.956387 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:35.955235 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0c05c4b-da86-4622-88ec-960248babce5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c05c4b-da86-4622-88ec-960248babce5-openshift-service-ca\") pod \"e0c05c4b-da86-4622-88ec-960248babce5\" (UID: \"e0c05c4b-da86-4622-88ec-960248babce5\") " Oct 30 18:38:35.956387 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:35.955347 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0c05c4b-da86-4622-88ec-960248babce5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c05c4b-da86-4622-88ec-960248babce5-maas-api-tls\") pod \"e0c05c4b-da86-4622-88ec-960248babce5\" (UID: \"e0c05c4b-da86-4622-88ec-960248babce5\") " Oct 30 18:38:35.956387 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:35.955436 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c05c4b-da86-4622-88ec-960248babce5-kube-api-access-h7275\" (UniqueName: \"kubernetes.io/projected/e0c05c4b-da86-4622-88ec-960248babce5-kube-api-access-h7275\") pod \"e0c05c4b-da86-4622-88ec-960248babce5\" (UID: \"e0c05c4b-da86-4622-88ec-960248babce5\") " Oct 30 18:38:35.956387 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:35.956108 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c05c4b-da86-4622-88ec-960248babce5-openshift-service-ca" pod "e0c05c4b-da86-4622-88ec-960248babce5" (UID: "e0c05c4b-da86-4622-88ec-960248babce5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:35.957436 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:35.957143 4603 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c05c4b-da86-4622-88ec-960248babce5-openshift-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:35.982237 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:35.963206 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c05c4b-da86-4622-88ec-960248babce5-maas-api-tls" pod "e0c05c4b-da86-4622-88ec-960248babce5" (UID: "e0c05c4b-da86-4622-88ec-960248babce5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:35.982237 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:35.971662 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c05c4b-da86-4622-88ec-960248babce5-kube-api-access-h7275" pod "e0c05c4b-da86-4622-88ec-960248babce5" (UID: "e0c05c4b-da86-4622-88ec-960248babce5"). InnerVolumeSpecName "kube-api-access-h7275". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:36.055530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:36.049081 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c05c4b-da86-4622-88ec-960248babce5-metrics-certs" pod "e0c05c4b-da86-4622-88ec-960248babce5" (UID: "e0c05c4b-da86-4622-88ec-960248babce5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:36.074526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:36.072945 4603 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c05c4b-da86-4622-88ec-960248babce5-maas-api-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:36.074526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:36.072987 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7275\" (UniqueName: \"kubernetes.io/projected/e0c05c4b-da86-4622-88ec-960248babce5-kube-api-access-h7275\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:36.074526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:36.073003 4603 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0c05c4b-da86-4622-88ec-960248babce5-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:36.312813 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:36.312778 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2"] Oct 30 18:38:36.338296 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:36.338260 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-57ee15-59946b47b6-djgd2"] Oct 30 18:38:36.467318 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:36.467281 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:36.719847 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:36.719815 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:37.175603 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:37.175567 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:37.410515 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:37.410475 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0c05c4b-da86-4622-88ec-960248babce5" path="/var/lib/kubelet/pods/e0c05c4b-da86-4622-88ec-960248babce5/volumes" Oct 30 18:38:38.817941 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:38.817910 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:39.042265 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:39.042212 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:40.915935 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:40.915901 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:41.553144 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:41.552378 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:41.987806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:41.987701 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:42.340292 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:42.340261 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:42.553919 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:42.553882 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:43.548014 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:43.547971 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:43.821429 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:43.821343 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w"] Oct 30 18:38:43.821629 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:43.821609 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" podUID="d79321de-091c-47e7-8320-3471ef8f554b" containerName="maas-api" containerID="cri-o://409b7a0d2a7e09c9d744649318afc46121ab514d766b2e59632990c2d94d1b00" gracePeriod=30 Oct 30 18:38:43.840606 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:43.840577 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:44.019996 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:44.019935 4603 generic.go:356] "Generic (PLEG): container finished" podID="d79321de-091c-47e7-8320-3471ef8f554b" containerID="409b7a0d2a7e09c9d744649318afc46121ab514d766b2e59632990c2d94d1b00" exitCode=0 Oct 30 18:38:44.020272 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:44.020093 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" event={"ID":"d79321de-091c-47e7-8320-3471ef8f554b","Type":"ContainerDied","Data":"409b7a0d2a7e09c9d744649318afc46121ab514d766b2e59632990c2d94d1b00"} Oct 30 18:38:44.038410 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:44.038378 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:44.091147 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:44.091093 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" Oct 30 18:38:44.257199 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:44.257166 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d79321de-091c-47e7-8320-3471ef8f554b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79321de-091c-47e7-8320-3471ef8f554b-metrics-certs\") pod \"d79321de-091c-47e7-8320-3471ef8f554b\" (UID: \"d79321de-091c-47e7-8320-3471ef8f554b\") " Oct 30 18:38:44.257382 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:44.257223 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d79321de-091c-47e7-8320-3471ef8f554b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79321de-091c-47e7-8320-3471ef8f554b-openshift-service-ca\") pod \"d79321de-091c-47e7-8320-3471ef8f554b\" (UID: \"d79321de-091c-47e7-8320-3471ef8f554b\") " Oct 30 18:38:44.257382 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:44.257264 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d79321de-091c-47e7-8320-3471ef8f554b-kube-api-access-2lstz\" (UniqueName: \"kubernetes.io/projected/d79321de-091c-47e7-8320-3471ef8f554b-kube-api-access-2lstz\") pod \"d79321de-091c-47e7-8320-3471ef8f554b\" (UID: \"d79321de-091c-47e7-8320-3471ef8f554b\") " Oct 30 18:38:44.257382 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:44.257283 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d79321de-091c-47e7-8320-3471ef8f554b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d79321de-091c-47e7-8320-3471ef8f554b-maas-api-tls\") pod \"d79321de-091c-47e7-8320-3471ef8f554b\" (UID: \"d79321de-091c-47e7-8320-3471ef8f554b\") " Oct 30 18:38:44.257698 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:44.257671 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d79321de-091c-47e7-8320-3471ef8f554b-openshift-service-ca" pod "d79321de-091c-47e7-8320-3471ef8f554b" (UID: "d79321de-091c-47e7-8320-3471ef8f554b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:44.258298 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:44.258272 4603 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79321de-091c-47e7-8320-3471ef8f554b-openshift-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:44.259680 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:44.259647 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d79321de-091c-47e7-8320-3471ef8f554b-maas-api-tls" pod "d79321de-091c-47e7-8320-3471ef8f554b" (UID: "d79321de-091c-47e7-8320-3471ef8f554b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:44.259790 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:44.259679 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d79321de-091c-47e7-8320-3471ef8f554b-metrics-certs" pod "d79321de-091c-47e7-8320-3471ef8f554b" (UID: "d79321de-091c-47e7-8320-3471ef8f554b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:44.259841 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:44.259799 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d79321de-091c-47e7-8320-3471ef8f554b-kube-api-access-2lstz" pod "d79321de-091c-47e7-8320-3471ef8f554b" (UID: "d79321de-091c-47e7-8320-3471ef8f554b"). InnerVolumeSpecName "kube-api-access-2lstz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:44.380592 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:44.380405 4603 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79321de-091c-47e7-8320-3471ef8f554b-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:44.380592 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:44.380443 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lstz\" (UniqueName: \"kubernetes.io/projected/d79321de-091c-47e7-8320-3471ef8f554b-kube-api-access-2lstz\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:44.380592 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:44.380459 4603 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d79321de-091c-47e7-8320-3471ef8f554b-maas-api-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:44.639584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:44.638685 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:45.042528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:45.040406 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:45.065882 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:45.064908 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" event={"ID":"d79321de-091c-47e7-8320-3471ef8f554b","Type":"ContainerDied","Data":"02dd29294222bb11e0e173bf72d7d700549565cd00e7fc50010d9608725c1c70"} Oct 30 18:38:45.065882 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:45.064989 4603 scope.go:122] "RemoveContainer" containerID="409b7a0d2a7e09c9d744649318afc46121ab514d766b2e59632990c2d94d1b00" Oct 30 18:38:45.065882 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:45.065223 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w" Oct 30 18:38:45.131473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:45.131441 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w"] Oct 30 18:38:45.150376 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:45.150341 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-58d08b-55cd849dcd-9gt2w"] Oct 30 18:38:45.315138 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:45.315057 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:45.439890 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:45.439857 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d79321de-091c-47e7-8320-3471ef8f554b" path="/var/lib/kubelet/pods/d79321de-091c-47e7-8320-3471ef8f554b/volumes" Oct 30 18:38:47.683493 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:47.683464 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:48.472513 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:48.472469 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:48.864880 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:48.864831 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:48.958000 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:48.957967 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp"] Oct 30 18:38:48.958227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:48.958206 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" podUID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerName="istio-proxy" containerID="cri-o://512a4781daf6c9dda86706fb8577f67604b2aa0887f8a061d5ba0f82bf8be56f" gracePeriod=30 Oct 30 18:38:48.969443 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:48.969416 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7"] Oct 30 18:38:48.969681 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:48.969659 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" podUID="b6e43478-8cc1-4452-8bc8-fed14f127896" containerName="istio-proxy" containerID="cri-o://3d67718133ad27c33c64da0b59e48736a8e2c418167704665f1bd379f42c597c" gracePeriod=30 Oct 30 18:38:49.029530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:49.029239 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk"] Oct 30 18:38:49.029530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:49.029474 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" podUID="187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" containerName="istio-proxy" containerID="cri-o://268d83b09abb47512048569aaa77e68400eea8df484d57fcd9dbd35e1fa8aed9" gracePeriod=30 Oct 30 18:38:49.081264 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:49.069391 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j"] Oct 30 18:38:49.081264 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:49.069735 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" podUID="ec3c95aa-215c-4f15-8dc8-bc0beb954236" containerName="istio-proxy" containerID="cri-o://6d25961fc4da23bdd7bf8c4c12a2971d7f2498c46d49eb1ace177ff767db29ce" gracePeriod=30 Oct 30 18:38:49.228595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:49.228192 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:49.983142 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:49.981749 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:50.324181 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:50.324143 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:50.571045 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:50.571004 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:51.669788 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:51.669306 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:51.671150 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:51.671125 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn"] Oct 30 18:38:51.679970 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:51.679905 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" podUID="cb49f732-be41-4ed9-b7e6-a4eb39e146e2" containerName="maas-api" containerID="cri-o://248b7da02d2d28841419f3fabadca21412e503c09e1f391aa5491fd86823f0c3" gracePeriod=30 Oct 30 18:38:51.861514 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:51.861464 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:51.946363 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:38:51.946330 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:38:51.986753 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:51.986722 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" Oct 30 18:38:52.026405 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.026371 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv"] Oct 30 18:38:52.026626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.026602 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" podUID="d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" containerName="istio-proxy" containerID="cri-o://e4858c1f49fde3ecb3fcbc8a2e63832ecbbdc6ce466a3e4a2a8f0919fdd12eeb" gracePeriod=30 Oct 30 18:38:52.046348 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.046316 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc"] Oct 30 18:38:52.046608 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.046583 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" podUID="f7882513-ef6d-4b11-871e-39a29ad022df" containerName="istio-proxy" containerID="cri-o://9098652c215ffeb76c0a5f8ae2d3f9069fbb61ceb96ec93ee5adcb36dc2a401d" gracePeriod=30 Oct 30 18:38:52.063521 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.063476 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj"] Oct 30 18:38:52.063790 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.063758 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" podUID="58e2469b-dc0d-40ab-bd74-a47c4a9e438b" containerName="istio-proxy" containerID="cri-o://113c96e7649a6ae6d7ea602cfcf4fa522f5e7e5fdf3d1fe4107a9ff9456ea918" gracePeriod=30 Oct 30 18:38:52.064947 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.064920 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-metrics-certs\") pod \"cb49f732-be41-4ed9-b7e6-a4eb39e146e2\" (UID: \"cb49f732-be41-4ed9-b7e6-a4eb39e146e2\") " Oct 30 18:38:52.065151 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.065131 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-maas-api-tls\") pod \"cb49f732-be41-4ed9-b7e6-a4eb39e146e2\" (UID: \"cb49f732-be41-4ed9-b7e6-a4eb39e146e2\") " Oct 30 18:38:52.065235 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.065199 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-openshift-service-ca\") pod \"cb49f732-be41-4ed9-b7e6-a4eb39e146e2\" (UID: \"cb49f732-be41-4ed9-b7e6-a4eb39e146e2\") " Oct 30 18:38:52.066570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.065385 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-kube-api-access-vplnb\" (UniqueName: \"kubernetes.io/projected/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-kube-api-access-vplnb\") pod \"cb49f732-be41-4ed9-b7e6-a4eb39e146e2\" (UID: \"cb49f732-be41-4ed9-b7e6-a4eb39e146e2\") " Oct 30 18:38:52.066570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.066490 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-openshift-service-ca" pod "cb49f732-be41-4ed9-b7e6-a4eb39e146e2" (UID: "cb49f732-be41-4ed9-b7e6-a4eb39e146e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:52.067068 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.066996 4603 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-openshift-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:52.069008 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.068979 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v"] Oct 30 18:38:52.069121 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.069055 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-maas-api-tls" pod "cb49f732-be41-4ed9-b7e6-a4eb39e146e2" (UID: "cb49f732-be41-4ed9-b7e6-a4eb39e146e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:52.069254 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.069226 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" podUID="5411300d-c1aa-4397-be3d-0ce869a1c1cb" containerName="istio-proxy" containerID="cri-o://d3651a6683955db5bd115cb4b16bb4ff672c158d35fd8f25cf04586a82ad9e4b" gracePeriod=30 Oct 30 18:38:52.069693 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.069666 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-kube-api-access-vplnb" pod "cb49f732-be41-4ed9-b7e6-a4eb39e146e2" (UID: "cb49f732-be41-4ed9-b7e6-a4eb39e146e2"). InnerVolumeSpecName "kube-api-access-vplnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:52.069782 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.069695 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-metrics-certs" pod "cb49f732-be41-4ed9-b7e6-a4eb39e146e2" (UID: "cb49f732-be41-4ed9-b7e6-a4eb39e146e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:52.279230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.203740 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vplnb\" (UniqueName: \"kubernetes.io/projected/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-kube-api-access-vplnb\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:52.279230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.203776 4603 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:52.279230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.203791 4603 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb49f732-be41-4ed9-b7e6-a4eb39e146e2-maas-api-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:52.279230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.233185 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:52.279230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.279106 4603 generic.go:356] "Generic (PLEG): container finished" podID="cb49f732-be41-4ed9-b7e6-a4eb39e146e2" containerID="248b7da02d2d28841419f3fabadca21412e503c09e1f391aa5491fd86823f0c3" exitCode=0 Oct 30 18:38:52.281156 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.279676 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" Oct 30 18:38:52.281156 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.279901 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" event={"ID":"cb49f732-be41-4ed9-b7e6-a4eb39e146e2","Type":"ContainerDied","Data":"248b7da02d2d28841419f3fabadca21412e503c09e1f391aa5491fd86823f0c3"} Oct 30 18:38:52.281156 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.279978 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn" event={"ID":"cb49f732-be41-4ed9-b7e6-a4eb39e146e2","Type":"ContainerDied","Data":"e060def9e347242dc2913da474615dd78715d578a2a1040e92c5057507a55245"} Oct 30 18:38:52.281156 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.280013 4603 scope.go:122] "RemoveContainer" containerID="248b7da02d2d28841419f3fabadca21412e503c09e1f391aa5491fd86823f0c3" Oct 30 18:38:52.340655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.339325 4603 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:38:52.340655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.339383 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" podUID="58e2469b-dc0d-40ab-bd74-a47c4a9e438b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:38:52.392054 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.387463 4603 patch_prober.go:29] interesting pod/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:38:52.392054 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.387552 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" podUID="d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:52.524574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.522631 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn"] Oct 30 18:38:52.539654 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.537846 4603 scope.go:122] "RemoveContainer" containerID="248b7da02d2d28841419f3fabadca21412e503c09e1f391aa5491fd86823f0c3" Oct 30 18:38:52.539654 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:38:52.538254 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"248b7da02d2d28841419f3fabadca21412e503c09e1f391aa5491fd86823f0c3\": container with ID starting with 248b7da02d2d28841419f3fabadca21412e503c09e1f391aa5491fd86823f0c3 not found: ID does not exist" containerID="248b7da02d2d28841419f3fabadca21412e503c09e1f391aa5491fd86823f0c3" Oct 30 18:38:52.539654 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.538281 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"248b7da02d2d28841419f3fabadca21412e503c09e1f391aa5491fd86823f0c3"} err="failed to get container status \"248b7da02d2d28841419f3fabadca21412e503c09e1f391aa5491fd86823f0c3\": rpc error: code = NotFound desc = could not find container \"248b7da02d2d28841419f3fabadca21412e503c09e1f391aa5491fd86823f0c3\": container with ID starting with 248b7da02d2d28841419f3fabadca21412e503c09e1f391aa5491fd86823f0c3 not found: ID does not exist" Oct 30 18:38:52.554688 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.543725 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a1a31f-8bfbc58fc-2n5vn"] Oct 30 18:38:52.567251 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.567221 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:52.595762 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:38:52.595723 4603 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb49f732_be41_4ed9_b7e6_a4eb39e146e2.slice/crio-e060def9e347242dc2913da474615dd78715d578a2a1040e92c5057507a55245\": RecentStats: unable to find data in memory cache]" Oct 30 18:38:52.857208 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:52.857054 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:53.114786 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:53.114708 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:53.411767 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:53.411699 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb49f732-be41-4ed9-b7e6-a4eb39e146e2" path="/var/lib/kubelet/pods/cb49f732-be41-4ed9-b7e6-a4eb39e146e2/volumes" Oct 30 18:38:54.313305 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.313268 4603 generic.go:356] "Generic (PLEG): container finished" podID="ec3c95aa-215c-4f15-8dc8-bc0beb954236" containerID="6d25961fc4da23bdd7bf8c4c12a2971d7f2498c46d49eb1ace177ff767db29ce" exitCode=0 Oct 30 18:38:54.313771 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.313427 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" event={"ID":"ec3c95aa-215c-4f15-8dc8-bc0beb954236","Type":"ContainerDied","Data":"6d25961fc4da23bdd7bf8c4c12a2971d7f2498c46d49eb1ace177ff767db29ce"} Oct 30 18:38:54.316676 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.316647 4603 generic.go:356] "Generic (PLEG): container finished" podID="187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" containerID="268d83b09abb47512048569aaa77e68400eea8df484d57fcd9dbd35e1fa8aed9" exitCode=0 Oct 30 18:38:54.316807 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.316777 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" event={"ID":"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e","Type":"ContainerDied","Data":"268d83b09abb47512048569aaa77e68400eea8df484d57fcd9dbd35e1fa8aed9"} Oct 30 18:38:54.320361 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.320332 4603 generic.go:356] "Generic (PLEG): container finished" podID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerID="512a4781daf6c9dda86706fb8577f67604b2aa0887f8a061d5ba0f82bf8be56f" exitCode=0 Oct 30 18:38:54.320485 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.320462 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" event={"ID":"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a","Type":"ContainerDied","Data":"512a4781daf6c9dda86706fb8577f67604b2aa0887f8a061d5ba0f82bf8be56f"} Oct 30 18:38:54.320590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.320534 4603 scope.go:122] "RemoveContainer" containerID="eaa1fb3ae846f7041b3a2811b2d48bc6aaaeeaf45ef5e5873ac44693145d9a66" Oct 30 18:38:54.331221 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.326970 4603 generic.go:356] "Generic (PLEG): container finished" podID="b6e43478-8cc1-4452-8bc8-fed14f127896" containerID="3d67718133ad27c33c64da0b59e48736a8e2c418167704665f1bd379f42c597c" exitCode=0 Oct 30 18:38:54.331221 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.327014 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" event={"ID":"b6e43478-8cc1-4452-8bc8-fed14f127896","Type":"ContainerDied","Data":"3d67718133ad27c33c64da0b59e48736a8e2c418167704665f1bd379f42c597c"} Oct 30 18:38:54.338091 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.338069 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:38:54.370600 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.369853 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:38:54.373480 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.373455 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:38:54.451139 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.451113 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:38:54.479237 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.479208 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-envoy\") pod \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " Oct 30 18:38:54.479402 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.479253 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-podinfo\") pod \"b6e43478-8cc1-4452-8bc8-fed14f127896\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " Oct 30 18:38:54.479402 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.479292 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-data\") pod \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " Oct 30 18:38:54.479402 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.479328 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istiod-ca-cert\") pod \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " Oct 30 18:38:54.479402 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.479353 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6e43478-8cc1-4452-8bc8-fed14f127896-kube-api-access-wxwjm\" (UniqueName: \"kubernetes.io/projected/b6e43478-8cc1-4452-8bc8-fed14f127896-kube-api-access-wxwjm\") pod \"b6e43478-8cc1-4452-8bc8-fed14f127896\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " Oct 30 18:38:54.479402 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.479380 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-token\" (UniqueName: \"kubernetes.io/projected/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-token\") pod \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " Oct 30 18:38:54.480428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.479407 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-token\" (UniqueName: \"kubernetes.io/projected/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-token\") pod \"b6e43478-8cc1-4452-8bc8-fed14f127896\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " Oct 30 18:38:54.480428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.479442 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-workload-socket\") pod \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " Oct 30 18:38:54.480428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.479474 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-podinfo\") pod \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " Oct 30 18:38:54.480428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.479537 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-data\") pod \"b6e43478-8cc1-4452-8bc8-fed14f127896\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " Oct 30 18:38:54.480428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.479574 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-credential-socket\") pod \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " Oct 30 18:38:54.480428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.479606 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-credential-socket\") pod \"b6e43478-8cc1-4452-8bc8-fed14f127896\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " Oct 30 18:38:54.480428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.479662 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-workload-certs\") pod \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " Oct 30 18:38:54.480428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.479749 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-token\" (UniqueName: \"kubernetes.io/projected/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-token\") pod \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " Oct 30 18:38:54.480428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.479774 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-workload-certs\") pod \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " Oct 30 18:38:54.480428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.479818 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-envoy\") pod \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " Oct 30 18:38:54.480428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.479927 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-workload-socket\") pod \"b6e43478-8cc1-4452-8bc8-fed14f127896\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " Oct 30 18:38:54.480428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.480006 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-kube-api-access-9jl4z\" (UniqueName: \"kubernetes.io/projected/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-kube-api-access-9jl4z\") pod \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " Oct 30 18:38:54.480428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.480119 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istiod-ca-cert\") pod \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " Oct 30 18:38:54.482996 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.482789 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-workload-certs" pod "187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" (UID: "187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.483348 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.483156 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-credential-socket" pod "b6e43478-8cc1-4452-8bc8-fed14f127896" (UID: "b6e43478-8cc1-4452-8bc8-fed14f127896"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.484013 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.483613 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istiod-ca-cert" pod "187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" (UID: "187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:54.484440 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.484165 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-workload-certs" pod "744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" (UID: "744e9a72-c23c-4b8f-b0fa-ef7f791ce58a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.484877 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.484855 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-credential-socket" pod "744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" (UID: "744e9a72-c23c-4b8f-b0fa-ef7f791ce58a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.484990 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.484958 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-workload-certs\") pod \"b6e43478-8cc1-4452-8bc8-fed14f127896\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " Oct 30 18:38:54.485176 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.485156 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6e43478-8cc1-4452-8bc8-fed14f127896-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6e43478-8cc1-4452-8bc8-fed14f127896-istiod-ca-cert\") pod \"b6e43478-8cc1-4452-8bc8-fed14f127896\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " Oct 30 18:38:54.485360 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.485342 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-data\") pod \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " Oct 30 18:38:54.485582 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.485563 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-kube-api-access-xr8gp\" (UniqueName: \"kubernetes.io/projected/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-kube-api-access-xr8gp\") pod \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\" (UID: \"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a\") " Oct 30 18:38:54.485760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.485744 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-envoy\") pod \"b6e43478-8cc1-4452-8bc8-fed14f127896\" (UID: \"b6e43478-8cc1-4452-8bc8-fed14f127896\") " Oct 30 18:38:54.485930 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.485913 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-podinfo\") pod \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " Oct 30 18:38:54.486105 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.486088 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-credential-socket\") pod \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " Oct 30 18:38:54.486278 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.486261 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-workload-socket\") pod \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\" (UID: \"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e\") " Oct 30 18:38:54.487132 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.487111 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.487357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.487138 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.487357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.487151 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.487357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.487165 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.487357 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.487179 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.488608 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.485358 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-workload-certs" pod "b6e43478-8cc1-4452-8bc8-fed14f127896" (UID: "b6e43478-8cc1-4452-8bc8-fed14f127896"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.488608 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.485649 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-data" pod "744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" (UID: "744e9a72-c23c-4b8f-b0fa-ef7f791ce58a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.488775 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.485860 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-workload-socket" pod "b6e43478-8cc1-4452-8bc8-fed14f127896" (UID: "b6e43478-8cc1-4452-8bc8-fed14f127896"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.488775 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.486148 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istiod-ca-cert" pod "744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" (UID: "744e9a72-c23c-4b8f-b0fa-ef7f791ce58a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:54.488775 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.486467 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e43478-8cc1-4452-8bc8-fed14f127896-istiod-ca-cert" pod "b6e43478-8cc1-4452-8bc8-fed14f127896" (UID: "b6e43478-8cc1-4452-8bc8-fed14f127896"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:54.488775 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.487333 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-data" pod "187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" (UID: "187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.488775 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.487361 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-workload-socket" pod "187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" (UID: "187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.488775 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.488637 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-workload-socket" pod "744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" (UID: "744e9a72-c23c-4b8f-b0fa-ef7f791ce58a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.489044 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.488819 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-data" pod "b6e43478-8cc1-4452-8bc8-fed14f127896" (UID: "b6e43478-8cc1-4452-8bc8-fed14f127896"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.489044 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.488848 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-token" pod "187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" (UID: "187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:54.489044 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.488977 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-token" pod "744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" (UID: "744e9a72-c23c-4b8f-b0fa-ef7f791ce58a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:54.489044 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.488986 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-credential-socket" pod "187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" (UID: "187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.490327 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.490303 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-podinfo" pod "744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" (UID: "744e9a72-c23c-4b8f-b0fa-ef7f791ce58a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:38:54.491600 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.491548 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-envoy" pod "187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" (UID: "187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.491785 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.491766 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-podinfo" pod "187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" (UID: "187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:38:54.492956 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.492917 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-envoy" pod "744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" (UID: "744e9a72-c23c-4b8f-b0fa-ef7f791ce58a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.493030 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.492961 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-kube-api-access-9jl4z" pod "187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" (UID: "187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e"). InnerVolumeSpecName "kube-api-access-9jl4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:54.493386 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.493321 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-envoy" pod "b6e43478-8cc1-4452-8bc8-fed14f127896" (UID: "b6e43478-8cc1-4452-8bc8-fed14f127896"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.493386 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.493359 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e43478-8cc1-4452-8bc8-fed14f127896-kube-api-access-wxwjm" pod "b6e43478-8cc1-4452-8bc8-fed14f127896" (UID: "b6e43478-8cc1-4452-8bc8-fed14f127896"). InnerVolumeSpecName "kube-api-access-wxwjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:54.493386 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.493364 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-podinfo" pod "b6e43478-8cc1-4452-8bc8-fed14f127896" (UID: "b6e43478-8cc1-4452-8bc8-fed14f127896"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:38:54.494140 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.494115 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-kube-api-access-xr8gp" pod "744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" (UID: "744e9a72-c23c-4b8f-b0fa-ef7f791ce58a"). InnerVolumeSpecName "kube-api-access-xr8gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:54.495273 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.495253 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-token" pod "b6e43478-8cc1-4452-8bc8-fed14f127896" (UID: "b6e43478-8cc1-4452-8bc8-fed14f127896"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:54.589289 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.589254 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-envoy\") pod \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " Oct 30 18:38:54.589623 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.589604 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-token\" (UniqueName: \"kubernetes.io/projected/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-token\") pod \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " Oct 30 18:38:54.589778 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.589763 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-podinfo\") pod \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " Oct 30 18:38:54.590070 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.590052 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec3c95aa-215c-4f15-8dc8-bc0beb954236-kube-api-access-gg4nr\" (UniqueName: \"kubernetes.io/projected/ec3c95aa-215c-4f15-8dc8-bc0beb954236-kube-api-access-gg4nr\") pod \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " Oct 30 18:38:54.590224 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.590211 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-credential-socket\") pod \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " Oct 30 18:38:54.590373 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.590357 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-data\") pod \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " Oct 30 18:38:54.590481 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.590468 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istiod-ca-cert\") pod \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " Oct 30 18:38:54.590664 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.590649 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-workload-socket\") pod \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " Oct 30 18:38:54.590873 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.590855 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-workload-certs\") pod \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\" (UID: \"ec3c95aa-215c-4f15-8dc8-bc0beb954236\") " Oct 30 18:38:54.591807 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.591785 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.591914 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.591898 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.592024 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.591920 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.592024 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.591933 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.592024 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.591948 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxwjm\" (UniqueName: \"kubernetes.io/projected/b6e43478-8cc1-4452-8bc8-fed14f127896-kube-api-access-wxwjm\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.592024 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.591962 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.592024 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.591976 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.592024 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.591988 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.592024 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.592002 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.592024 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.592016 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.592963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.592029 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.592963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.592043 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.592963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.592058 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.592963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.592073 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jl4z\" (UniqueName: \"kubernetes.io/projected/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-kube-api-access-9jl4z\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.592963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.592087 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.592963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.592101 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.592963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.592113 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6e43478-8cc1-4452-8bc8-fed14f127896-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.592963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.592128 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.592963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.592141 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr8gp\" (UniqueName: \"kubernetes.io/projected/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a-kube-api-access-xr8gp\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.592963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.592154 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6e43478-8cc1-4452-8bc8-fed14f127896-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.592963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.592169 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.592963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.592182 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.592963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.592612 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-data" pod "ec3c95aa-215c-4f15-8dc8-bc0beb954236" (UID: "ec3c95aa-215c-4f15-8dc8-bc0beb954236"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.592963 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.592799 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istiod-ca-cert" pod "ec3c95aa-215c-4f15-8dc8-bc0beb954236" (UID: "ec3c95aa-215c-4f15-8dc8-bc0beb954236"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:54.594564 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.593062 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-workload-socket" pod "ec3c95aa-215c-4f15-8dc8-bc0beb954236" (UID: "ec3c95aa-215c-4f15-8dc8-bc0beb954236"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.594564 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.593249 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-workload-certs" pod "ec3c95aa-215c-4f15-8dc8-bc0beb954236" (UID: "ec3c95aa-215c-4f15-8dc8-bc0beb954236"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.595480 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.595452 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-credential-socket" pod "ec3c95aa-215c-4f15-8dc8-bc0beb954236" (UID: "ec3c95aa-215c-4f15-8dc8-bc0beb954236"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.597816 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.597769 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3c95aa-215c-4f15-8dc8-bc0beb954236-kube-api-access-gg4nr" pod "ec3c95aa-215c-4f15-8dc8-bc0beb954236" (UID: "ec3c95aa-215c-4f15-8dc8-bc0beb954236"). InnerVolumeSpecName "kube-api-access-gg4nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:54.603798 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.603777 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-envoy" pod "ec3c95aa-215c-4f15-8dc8-bc0beb954236" (UID: "ec3c95aa-215c-4f15-8dc8-bc0beb954236"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.605227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.605191 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-podinfo" pod "ec3c95aa-215c-4f15-8dc8-bc0beb954236" (UID: "ec3c95aa-215c-4f15-8dc8-bc0beb954236"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:38:54.605682 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.605649 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-token" pod "ec3c95aa-215c-4f15-8dc8-bc0beb954236" (UID: "ec3c95aa-215c-4f15-8dc8-bc0beb954236"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:54.693444 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.693416 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.700648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.693448 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.700648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.693457 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.700648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.693467 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.700648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.693475 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg4nr\" (UniqueName: \"kubernetes.io/projected/ec3c95aa-215c-4f15-8dc8-bc0beb954236-kube-api-access-gg4nr\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.700648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.693483 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.700648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.693492 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.700648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.693535 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3c95aa-215c-4f15-8dc8-bc0beb954236-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.700648 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.693544 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3c95aa-215c-4f15-8dc8-bc0beb954236-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.972271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:54.965057 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:55.335288 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.335259 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" Oct 30 18:38:55.335806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.335289 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7" event={"ID":"b6e43478-8cc1-4452-8bc8-fed14f127896","Type":"ContainerDied","Data":"e0179e7723ba80fe74d0626fb7e382b3b74a5f773a12acfd12eb0e36e537a7d7"} Oct 30 18:38:55.335806 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.335365 4603 scope.go:122] "RemoveContainer" containerID="3d67718133ad27c33c64da0b59e48736a8e2c418167704665f1bd379f42c597c" Oct 30 18:38:55.337045 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.337026 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" Oct 30 18:38:55.337161 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.337074 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j" event={"ID":"ec3c95aa-215c-4f15-8dc8-bc0beb954236","Type":"ContainerDied","Data":"973cb30f59f0578a083423919b6334e606a7d024744357145f7f1c1887b7b911"} Oct 30 18:38:55.338837 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.338817 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" Oct 30 18:38:55.338837 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.338826 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk" event={"ID":"187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e","Type":"ContainerDied","Data":"1344958c6607e3761f5e253a7cf4f0a9836ca2a84f8da57cb7d73c3fa81be914"} Oct 30 18:38:55.340714 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.340690 4603 generic.go:356] "Generic (PLEG): container finished" podID="d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" containerID="290e03c48d5b4e3be632e3fdaaa0e1f2be7841d7d786197ac96e93474e3cfe88" exitCode=0 Oct 30 18:38:55.340839 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.340761 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" event={"ID":"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3","Type":"ContainerDied","Data":"290e03c48d5b4e3be632e3fdaaa0e1f2be7841d7d786197ac96e93474e3cfe88"} Oct 30 18:38:55.341250 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.341221 4603 scope.go:122] "RemoveContainer" containerID="290e03c48d5b4e3be632e3fdaaa0e1f2be7841d7d786197ac96e93474e3cfe88" Oct 30 18:38:55.342277 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.342257 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" event={"ID":"744e9a72-c23c-4b8f-b0fa-ef7f791ce58a","Type":"ContainerDied","Data":"79f4922a74814972e3fc9332fa7feb944ca14db75955fcaa9d793cbc4d8d435b"} Oct 30 18:38:55.342387 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.342292 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp" Oct 30 18:38:55.352114 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.352084 4603 scope.go:122] "RemoveContainer" containerID="6d25961fc4da23bdd7bf8c4c12a2971d7f2498c46d49eb1ace177ff767db29ce" Oct 30 18:38:55.352304 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.352267 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:38:55.352430 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.352366 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:38:55.352430 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.352415 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:38:55.367599 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.367582 4603 scope.go:122] "RemoveContainer" containerID="268d83b09abb47512048569aaa77e68400eea8df484d57fcd9dbd35e1fa8aed9" Oct 30 18:38:55.377091 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.377051 4603 scope.go:122] "RemoveContainer" containerID="512a4781daf6c9dda86706fb8577f67604b2aa0887f8a061d5ba0f82bf8be56f" Oct 30 18:38:55.601964 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.601885 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7"] Oct 30 18:38:55.612881 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.612814 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-rd7m7"] Oct 30 18:38:55.712696 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.712663 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp"] Oct 30 18:38:55.723858 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.723830 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-lxxdp"] Oct 30 18:38:55.766538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.766495 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j"] Oct 30 18:38:55.771517 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.771465 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-vqk6j"] Oct 30 18:38:55.812904 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.812876 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk"] Oct 30 18:38:55.830540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:55.830481 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-fs8qk"] Oct 30 18:38:56.352185 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:56.352146 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" event={"ID":"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3","Type":"ContainerStarted","Data":"1d3849088b0aca81ade421007a9b49e8010931bc4e818b694e593426108fa8a2"} Oct 30 18:38:56.674880 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:56.674788 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:38:56.675068 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:56.675045 4603 patch_prober.go:29] interesting pod/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:38:56.675116 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:56.675097 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" podUID="d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:38:57.250684 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.227651 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:57.370258 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.369392 4603 generic.go:356] "Generic (PLEG): container finished" podID="58e2469b-dc0d-40ab-bd74-a47c4a9e438b" containerID="113c96e7649a6ae6d7ea602cfcf4fa522f5e7e5fdf3d1fe4107a9ff9456ea918" exitCode=0 Oct 30 18:38:57.370258 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.369535 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" event={"ID":"58e2469b-dc0d-40ab-bd74-a47c4a9e438b","Type":"ContainerDied","Data":"113c96e7649a6ae6d7ea602cfcf4fa522f5e7e5fdf3d1fe4107a9ff9456ea918"} Oct 30 18:38:57.372297 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.372258 4603 generic.go:356] "Generic (PLEG): container finished" podID="5411300d-c1aa-4397-be3d-0ce869a1c1cb" containerID="d3651a6683955db5bd115cb4b16bb4ff672c158d35fd8f25cf04586a82ad9e4b" exitCode=0 Oct 30 18:38:57.372457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.372397 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" event={"ID":"5411300d-c1aa-4397-be3d-0ce869a1c1cb","Type":"ContainerDied","Data":"d3651a6683955db5bd115cb4b16bb4ff672c158d35fd8f25cf04586a82ad9e4b"} Oct 30 18:38:57.372457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.372445 4603 scope.go:122] "RemoveContainer" containerID="8a93ed842ff210b1ca4d9ffaf50a7eee57466fc23ceed53b565d582dc39d3878" Oct 30 18:38:57.377571 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.377549 4603 generic.go:356] "Generic (PLEG): container finished" podID="f7882513-ef6d-4b11-871e-39a29ad022df" containerID="9098652c215ffeb76c0a5f8ae2d3f9069fbb61ceb96ec93ee5adcb36dc2a401d" exitCode=0 Oct 30 18:38:57.377796 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.377630 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" event={"ID":"f7882513-ef6d-4b11-871e-39a29ad022df","Type":"ContainerDied","Data":"9098652c215ffeb76c0a5f8ae2d3f9069fbb61ceb96ec93ee5adcb36dc2a401d"} Oct 30 18:38:57.380001 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.379973 4603 generic.go:356] "Generic (PLEG): container finished" podID="d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" containerID="e4858c1f49fde3ecb3fcbc8a2e63832ecbbdc6ce466a3e4a2a8f0919fdd12eeb" exitCode=0 Oct 30 18:38:57.381448 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.381429 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" event={"ID":"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5","Type":"ContainerDied","Data":"e4858c1f49fde3ecb3fcbc8a2e63832ecbbdc6ce466a3e4a2a8f0919fdd12eeb"} Oct 30 18:38:57.437297 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.435631 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" path="/var/lib/kubelet/pods/187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e/volumes" Oct 30 18:38:57.437297 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.436323 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" path="/var/lib/kubelet/pods/744e9a72-c23c-4b8f-b0fa-ef7f791ce58a/volumes" Oct 30 18:38:57.437297 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.436931 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6e43478-8cc1-4452-8bc8-fed14f127896" path="/var/lib/kubelet/pods/b6e43478-8cc1-4452-8bc8-fed14f127896/volumes" Oct 30 18:38:57.438245 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.438057 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec3c95aa-215c-4f15-8dc8-bc0beb954236" path="/var/lib/kubelet/pods/ec3c95aa-215c-4f15-8dc8-bc0beb954236/volumes" Oct 30 18:38:57.503528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.499969 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:38:57.548223 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.547694 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-istio-envoy\") pod \"f7882513-ef6d-4b11-871e-39a29ad022df\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " Oct 30 18:38:57.548420 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.548245 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-workload-certs\") pod \"f7882513-ef6d-4b11-871e-39a29ad022df\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " Oct 30 18:38:57.548567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.548525 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-workload-certs" pod "f7882513-ef6d-4b11-871e-39a29ad022df" (UID: "f7882513-ef6d-4b11-871e-39a29ad022df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.548870 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.548769 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7882513-ef6d-4b11-871e-39a29ad022df-istio-token\" (UniqueName: \"kubernetes.io/projected/f7882513-ef6d-4b11-871e-39a29ad022df-istio-token\") pod \"f7882513-ef6d-4b11-871e-39a29ad022df\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " Oct 30 18:38:57.549056 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.548982 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-credential-socket\") pod \"f7882513-ef6d-4b11-871e-39a29ad022df\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " Oct 30 18:38:57.549056 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.549034 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-workload-socket\") pod \"f7882513-ef6d-4b11-871e-39a29ad022df\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " Oct 30 18:38:57.549274 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.549248 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-istio-data\") pod \"f7882513-ef6d-4b11-871e-39a29ad022df\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " Oct 30 18:38:57.550041 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.549985 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7882513-ef6d-4b11-871e-39a29ad022df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7882513-ef6d-4b11-871e-39a29ad022df-istio-podinfo\") pod \"f7882513-ef6d-4b11-871e-39a29ad022df\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " Oct 30 18:38:57.550305 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.550268 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7882513-ef6d-4b11-871e-39a29ad022df-kube-api-access-8r6d9\" (UniqueName: \"kubernetes.io/projected/f7882513-ef6d-4b11-871e-39a29ad022df-kube-api-access-8r6d9\") pod \"f7882513-ef6d-4b11-871e-39a29ad022df\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " Oct 30 18:38:57.550473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.550445 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7882513-ef6d-4b11-871e-39a29ad022df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7882513-ef6d-4b11-871e-39a29ad022df-istiod-ca-cert\") pod \"f7882513-ef6d-4b11-871e-39a29ad022df\" (UID: \"f7882513-ef6d-4b11-871e-39a29ad022df\") " Oct 30 18:38:57.551833 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.549605 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-istio-data" pod "f7882513-ef6d-4b11-871e-39a29ad022df" (UID: "f7882513-ef6d-4b11-871e-39a29ad022df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.551833 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.550862 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-credential-socket" pod "f7882513-ef6d-4b11-871e-39a29ad022df" (UID: "f7882513-ef6d-4b11-871e-39a29ad022df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.551833 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.551010 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-workload-socket" pod "f7882513-ef6d-4b11-871e-39a29ad022df" (UID: "f7882513-ef6d-4b11-871e-39a29ad022df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.552679 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.552313 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.552679 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.552339 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.552679 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.552354 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.552679 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.552368 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.552679 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.552529 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7882513-ef6d-4b11-871e-39a29ad022df-istiod-ca-cert" pod "f7882513-ef6d-4b11-871e-39a29ad022df" (UID: "f7882513-ef6d-4b11-871e-39a29ad022df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:57.559229 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.557680 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7882513-ef6d-4b11-871e-39a29ad022df-istio-token" pod "f7882513-ef6d-4b11-871e-39a29ad022df" (UID: "f7882513-ef6d-4b11-871e-39a29ad022df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:57.559229 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.559116 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-istio-envoy" pod "f7882513-ef6d-4b11-871e-39a29ad022df" (UID: "f7882513-ef6d-4b11-871e-39a29ad022df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.559400 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.559246 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7882513-ef6d-4b11-871e-39a29ad022df-istio-podinfo" pod "f7882513-ef6d-4b11-871e-39a29ad022df" (UID: "f7882513-ef6d-4b11-871e-39a29ad022df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:38:57.559797 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.559773 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7882513-ef6d-4b11-871e-39a29ad022df-kube-api-access-8r6d9" pod "f7882513-ef6d-4b11-871e-39a29ad022df" (UID: "f7882513-ef6d-4b11-871e-39a29ad022df"). InnerVolumeSpecName "kube-api-access-8r6d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:57.642161 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.642120 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:38:57.643722 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.643699 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:38:57.648414 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.648392 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:38:57.654152 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.654126 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r6d9\" (UniqueName: \"kubernetes.io/projected/f7882513-ef6d-4b11-871e-39a29ad022df-kube-api-access-8r6d9\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.654247 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.654154 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7882513-ef6d-4b11-871e-39a29ad022df-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.654247 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.654169 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7882513-ef6d-4b11-871e-39a29ad022df-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.654247 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.654181 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7882513-ef6d-4b11-871e-39a29ad022df-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.654247 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.654193 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7882513-ef6d-4b11-871e-39a29ad022df-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.674633 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.674609 4603 patch_prober.go:29] interesting pod/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:38:57.674753 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.674660 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" podUID="d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:38:57.755344 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.755306 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-workload-socket\") pod \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " Oct 30 18:38:57.755599 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.755359 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-podinfo\") pod \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " Oct 30 18:38:57.755599 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.755404 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-credential-socket\") pod \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " Oct 30 18:38:57.755599 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.755459 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-envoy\") pod \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " Oct 30 18:38:57.755599 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.755553 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-workload-socket" pod "5411300d-c1aa-4397-be3d-0ce869a1c1cb" (UID: "5411300d-c1aa-4397-be3d-0ce869a1c1cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.756147 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.756122 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-workload-socket\") pod \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " Oct 30 18:38:57.756237 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.756170 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-envoy\") pod \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " Oct 30 18:38:57.756237 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.756128 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-credential-socket" pod "5411300d-c1aa-4397-be3d-0ce869a1c1cb" (UID: "5411300d-c1aa-4397-be3d-0ce869a1c1cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.756348 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.756205 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-workload-socket\") pod \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " Oct 30 18:38:57.756407 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.756346 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-workload-socket" pod "58e2469b-dc0d-40ab-bd74-a47c4a9e438b" (UID: "58e2469b-dc0d-40ab-bd74-a47c4a9e438b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.756407 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.756379 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-data\") pod \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " Oct 30 18:38:57.756407 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.756394 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-workload-socket" pod "d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" (UID: "d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.756621 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.756420 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-workload-certs\") pod \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " Oct 30 18:38:57.756621 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.756459 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-credential-socket\") pod \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " Oct 30 18:38:57.756621 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.756495 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-workload-certs\") pod \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " Oct 30 18:38:57.756621 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.756543 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istiod-ca-cert\") pod \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " Oct 30 18:38:57.756621 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.756573 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-credential-socket\") pod \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " Oct 30 18:38:57.756621 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.756605 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-data\") pod \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " Oct 30 18:38:57.756956 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.756633 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-data" pod "5411300d-c1aa-4397-be3d-0ce869a1c1cb" (UID: "5411300d-c1aa-4397-be3d-0ce869a1c1cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.756956 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.756642 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-token\" (UniqueName: \"kubernetes.io/projected/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-token\") pod \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " Oct 30 18:38:57.756956 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.756669 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-token\" (UniqueName: \"kubernetes.io/projected/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-token\") pod \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " Oct 30 18:38:57.756956 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.756715 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-data\") pod \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " Oct 30 18:38:57.756956 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.756773 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-workload-certs\") pod \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " Oct 30 18:38:57.756956 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.756812 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-workload-certs" pod "d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" (UID: "d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.756956 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.756825 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-workload-certs" pod "5411300d-c1aa-4397-be3d-0ce869a1c1cb" (UID: "5411300d-c1aa-4397-be3d-0ce869a1c1cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.756956 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.756872 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-podinfo\") pod \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " Oct 30 18:38:57.756956 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.756918 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-envoy\") pod \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " Oct 30 18:38:57.757435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.756993 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-token\" (UniqueName: \"kubernetes.io/projected/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-token\") pod \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " Oct 30 18:38:57.757435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.757022 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-credential-socket" pod "d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" (UID: "d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.757435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.757062 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5411300d-c1aa-4397-be3d-0ce869a1c1cb-kube-api-access-qqnzx\" (UniqueName: \"kubernetes.io/projected/5411300d-c1aa-4397-be3d-0ce869a1c1cb-kube-api-access-qqnzx\") pod \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " Oct 30 18:38:57.757435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.757077 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-workload-certs" pod "58e2469b-dc0d-40ab-bd74-a47c4a9e438b" (UID: "58e2469b-dc0d-40ab-bd74-a47c4a9e438b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.757435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.757092 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istiod-ca-cert\") pod \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " Oct 30 18:38:57.757435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.757139 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istiod-ca-cert\") pod \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " Oct 30 18:38:57.757435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.757202 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-kube-api-access-f6hrl\" (UniqueName: \"kubernetes.io/projected/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-kube-api-access-f6hrl\") pod \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\" (UID: \"58e2469b-dc0d-40ab-bd74-a47c4a9e438b\") " Oct 30 18:38:57.757435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.757272 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-podinfo\") pod \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\" (UID: \"5411300d-c1aa-4397-be3d-0ce869a1c1cb\") " Oct 30 18:38:57.757435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.757352 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-kube-api-access-qp9gf\" (UniqueName: \"kubernetes.io/projected/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-kube-api-access-qp9gf\") pod \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\" (UID: \"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5\") " Oct 30 18:38:57.757435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.757376 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istiod-ca-cert" pod "5411300d-c1aa-4397-be3d-0ce869a1c1cb" (UID: "5411300d-c1aa-4397-be3d-0ce869a1c1cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:57.759039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.758159 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.759039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.758182 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.759039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.758196 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.759039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.758212 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.759039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.758224 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.759039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.758238 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.759039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.758250 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.759039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.758262 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.759039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.758274 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.759039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.758287 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.759039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.758581 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-credential-socket" pod "58e2469b-dc0d-40ab-bd74-a47c4a9e438b" (UID: "58e2469b-dc0d-40ab-bd74-a47c4a9e438b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.759039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.758657 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-data" pod "58e2469b-dc0d-40ab-bd74-a47c4a9e438b" (UID: "58e2469b-dc0d-40ab-bd74-a47c4a9e438b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.759039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.758901 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-data" pod "d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" (UID: "d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.759039 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.758987 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istiod-ca-cert" pod "58e2469b-dc0d-40ab-bd74-a47c4a9e438b" (UID: "58e2469b-dc0d-40ab-bd74-a47c4a9e438b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:57.759941 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.759376 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istiod-ca-cert" pod "d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" (UID: "d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:57.760162 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.760115 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-kube-api-access-qp9gf" pod "d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" (UID: "d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5"). InnerVolumeSpecName "kube-api-access-qp9gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:57.760911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.760887 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-envoy" pod "d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" (UID: "d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.761334 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.761295 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-envoy" pod "5411300d-c1aa-4397-be3d-0ce869a1c1cb" (UID: "5411300d-c1aa-4397-be3d-0ce869a1c1cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.761553 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.761517 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-podinfo" pod "d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" (UID: "d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:38:57.763983 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.763950 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5411300d-c1aa-4397-be3d-0ce869a1c1cb-kube-api-access-qqnzx" pod "5411300d-c1aa-4397-be3d-0ce869a1c1cb" (UID: "5411300d-c1aa-4397-be3d-0ce869a1c1cb"). InnerVolumeSpecName "kube-api-access-qqnzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:57.764350 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.764200 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-podinfo" pod "5411300d-c1aa-4397-be3d-0ce869a1c1cb" (UID: "5411300d-c1aa-4397-be3d-0ce869a1c1cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:38:57.764350 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.764233 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-token" pod "d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" (UID: "d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:57.764350 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.764280 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-podinfo" pod "58e2469b-dc0d-40ab-bd74-a47c4a9e438b" (UID: "58e2469b-dc0d-40ab-bd74-a47c4a9e438b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:38:57.764350 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.764325 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-token" pod "58e2469b-dc0d-40ab-bd74-a47c4a9e438b" (UID: "58e2469b-dc0d-40ab-bd74-a47c4a9e438b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:57.764727 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.764708 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-envoy" pod "58e2469b-dc0d-40ab-bd74-a47c4a9e438b" (UID: "58e2469b-dc0d-40ab-bd74-a47c4a9e438b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.764982 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.764731 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-token" pod "5411300d-c1aa-4397-be3d-0ce869a1c1cb" (UID: "5411300d-c1aa-4397-be3d-0ce869a1c1cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:57.766172 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.766144 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-kube-api-access-f6hrl" pod "58e2469b-dc0d-40ab-bd74-a47c4a9e438b" (UID: "58e2469b-dc0d-40ab-bd74-a47c4a9e438b"). InnerVolumeSpecName "kube-api-access-f6hrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:57.859198 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.859169 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.859198 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.859196 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.859198 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.859205 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.859427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.859215 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqnzx\" (UniqueName: \"kubernetes.io/projected/5411300d-c1aa-4397-be3d-0ce869a1c1cb-kube-api-access-qqnzx\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.859427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.859223 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.859427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.859232 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.859427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.859240 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6hrl\" (UniqueName: \"kubernetes.io/projected/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-kube-api-access-f6hrl\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.859427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.859248 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.859427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.859256 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp9gf\" (UniqueName: \"kubernetes.io/projected/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-kube-api-access-qp9gf\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.859427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.859265 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.859427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.859273 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.859427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.859281 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.859427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.859289 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.859427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.859296 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e2469b-dc0d-40ab-bd74-a47c4a9e438b-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.859427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.859305 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.859427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.859312 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5411300d-c1aa-4397-be3d-0ce869a1c1cb-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.859427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:57.859319 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:38:58.135538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.135466 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:58.367493 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.367402 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:58.387306 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.387280 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" Oct 30 18:38:58.387717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.387311 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv" event={"ID":"d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5","Type":"ContainerDied","Data":"cf1280490bdcca624f1ad6dcf5504a4c321f645db2ee9932f33988affb14c367"} Oct 30 18:38:58.387717 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.387369 4603 scope.go:122] "RemoveContainer" containerID="e4858c1f49fde3ecb3fcbc8a2e63832ecbbdc6ce466a3e4a2a8f0919fdd12eeb" Oct 30 18:38:58.413528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.412758 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" event={"ID":"58e2469b-dc0d-40ab-bd74-a47c4a9e438b","Type":"ContainerDied","Data":"7f6390eee337d37c885cb4441e3136b4d2e6509de52f65d7f1c5fe5b67eb6f07"} Oct 30 18:38:58.413528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.412884 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj" Oct 30 18:38:58.426675 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.426650 4603 scope.go:122] "RemoveContainer" containerID="113c96e7649a6ae6d7ea602cfcf4fa522f5e7e5fdf3d1fe4107a9ff9456ea918" Oct 30 18:38:58.430545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.427747 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" Oct 30 18:38:58.430545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.427802 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v" event={"ID":"5411300d-c1aa-4397-be3d-0ce869a1c1cb","Type":"ContainerDied","Data":"1e1a2f77817654ef6ce18656efc4e90c8e00d64f32286ee8f8ec320afd72e736"} Oct 30 18:38:58.430545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.429979 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" event={"ID":"f7882513-ef6d-4b11-871e-39a29ad022df","Type":"ContainerDied","Data":"4f046068bb8e00146eb9331ed2895a7a39d3ea2480833e1c3f7f280e3a9ccf25"} Oct 30 18:38:58.430545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.430041 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc" Oct 30 18:38:58.438321 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.438306 4603 scope.go:122] "RemoveContainer" containerID="d3651a6683955db5bd115cb4b16bb4ff672c158d35fd8f25cf04586a82ad9e4b" Oct 30 18:38:58.447573 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.447556 4603 scope.go:122] "RemoveContainer" containerID="9098652c215ffeb76c0a5f8ae2d3f9069fbb61ceb96ec93ee5adcb36dc2a401d" Oct 30 18:38:58.529185 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.529146 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv"] Oct 30 18:38:58.543033 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.543005 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-8vfkv"] Oct 30 18:38:58.583704 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.583680 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj"] Oct 30 18:38:58.618574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.618448 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-kk9nj"] Oct 30 18:38:58.659899 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.659864 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v"] Oct 30 18:38:58.671276 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.671249 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-k8v7v"] Oct 30 18:38:58.674698 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.674674 4603 patch_prober.go:29] interesting pod/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:38:58.674811 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.674719 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" podUID="d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:38:58.721858 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.721829 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc"] Oct 30 18:38:58.726376 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:58.726350 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-hc6lc"] Oct 30 18:38:59.422823 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:59.422791 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5411300d-c1aa-4397-be3d-0ce869a1c1cb" path="/var/lib/kubelet/pods/5411300d-c1aa-4397-be3d-0ce869a1c1cb/volumes" Oct 30 18:38:59.423479 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:59.423458 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58e2469b-dc0d-40ab-bd74-a47c4a9e438b" path="/var/lib/kubelet/pods/58e2469b-dc0d-40ab-bd74-a47c4a9e438b/volumes" Oct 30 18:38:59.424017 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:59.423995 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" path="/var/lib/kubelet/pods/d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5/volumes" Oct 30 18:38:59.424846 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:59.424823 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7882513-ef6d-4b11-871e-39a29ad022df" path="/var/lib/kubelet/pods/f7882513-ef6d-4b11-871e-39a29ad022df/volumes" Oct 30 18:38:59.514060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:59.514018 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:59.674759 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:59.674678 4603 patch_prober.go:29] interesting pod/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:38:59.674759 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:38:59.674731 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" podUID="d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:39:00.675036 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:00.675002 4603 patch_prober.go:29] interesting pod/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:39:00.675471 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:00.675069 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" podUID="d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:39:01.361005 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:01.360974 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:01.675256 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:01.675167 4603 patch_prober.go:29] interesting pod/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:39:01.675709 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:01.675250 4603 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" podUID="d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:39:02.678838 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:02.678807 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:39:02.679342 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:02.679204 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:39:02.679820 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:02.679800 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:39:03.360388 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:03.360140 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:03.534667 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:03.534637 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:05.422147 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:05.422057 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:07.361723 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:07.361684 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:07.535794 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:07.535759 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:09.526214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:09.526159 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:11.066893 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:11.066859 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:11.266662 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:11.266629 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:11.557054 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:11.557020 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:11.858649 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:11.858553 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:13.559029 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:13.558994 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:15.389358 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:15.387149 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:17.069055 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:17.068569 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:18.045494 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:18.045460 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:18.539903 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:18.539874 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:18.765352 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:18.765298 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:19.258549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:19.258496 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:20.501827 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:20.501794 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:21.930636 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:39:21.930600 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:39:22.359574 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:22.359541 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:22.559549 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:22.559481 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:22.928220 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:22.928184 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:23.149943 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:23.149903 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:24.296869 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:24.294247 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:26.519382 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:26.519195 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:26.761616 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:26.761583 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:26.821315 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:26.821239 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf"] Oct 30 18:39:26.821541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:26.821512 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" podUID="d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" containerName="istio-proxy" containerID="cri-o://1d3849088b0aca81ade421007a9b49e8010931bc4e818b694e593426108fa8a2" gracePeriod=30 Oct 30 18:39:26.949538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:26.947926 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:28.793006 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:28.791423 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:30.502821 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:30.502777 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:31.848828 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:31.848797 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:32.095563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.093335 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:39:32.212485 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.212407 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-token\" (UniqueName: \"kubernetes.io/projected/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-token\") pod \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " Oct 30 18:39:32.212485 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.212484 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-data\") pod \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " Oct 30 18:39:32.213288 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.212544 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-workload-socket\") pod \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " Oct 30 18:39:32.213288 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.212709 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istiod-ca-cert\") pod \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " Oct 30 18:39:32.213288 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.212768 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-kube-api-access-hvcx4\" (UniqueName: \"kubernetes.io/projected/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-kube-api-access-hvcx4\") pod \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " Oct 30 18:39:32.213288 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.212832 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-credential-socket\") pod \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " Oct 30 18:39:32.213288 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.212943 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-envoy\") pod \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " Oct 30 18:39:32.213288 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.212987 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-workload-certs\") pod \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " Oct 30 18:39:32.213288 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.213002 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-workload-socket" pod "d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" (UID: "d5314716-efc8-4ce5-843f-fb4f5f2ba7e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:32.213288 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.213025 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-podinfo\") pod \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\" (UID: \"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3\") " Oct 30 18:39:32.213288 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.213259 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-workload-certs" pod "d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" (UID: "d5314716-efc8-4ce5-843f-fb4f5f2ba7e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:32.213861 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.213453 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-credential-socket" pod "d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" (UID: "d5314716-efc8-4ce5-843f-fb4f5f2ba7e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:32.213861 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.212909 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-data" pod "d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" (UID: "d5314716-efc8-4ce5-843f-fb4f5f2ba7e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:32.213861 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.213670 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:32.213861 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.213685 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:32.213861 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.213700 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:32.213861 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.213711 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:32.214180 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.213919 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istiod-ca-cert" pod "d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" (UID: "d5314716-efc8-4ce5-843f-fb4f5f2ba7e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:32.215477 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.215448 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-token" pod "d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" (UID: "d5314716-efc8-4ce5-843f-fb4f5f2ba7e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:32.215708 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.215688 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-envoy" pod "d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" (UID: "d5314716-efc8-4ce5-843f-fb4f5f2ba7e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:32.215966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.215947 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-kube-api-access-hvcx4" pod "d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" (UID: "d5314716-efc8-4ce5-843f-fb4f5f2ba7e3"). InnerVolumeSpecName "kube-api-access-hvcx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:32.216197 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.216177 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-podinfo" pod "d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" (UID: "d5314716-efc8-4ce5-843f-fb4f5f2ba7e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:39:32.315566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.315401 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:32.315566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.315449 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:32.315566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.315464 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvcx4\" (UniqueName: \"kubernetes.io/projected/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-kube-api-access-hvcx4\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:32.315566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.315479 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:32.315566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.315491 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:32.316114 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.315657 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:32.749251 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.749219 4603 generic.go:356] "Generic (PLEG): container finished" podID="d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" containerID="1d3849088b0aca81ade421007a9b49e8010931bc4e818b694e593426108fa8a2" exitCode=0 Oct 30 18:39:32.749439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.749266 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" Oct 30 18:39:32.749439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.749298 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" event={"ID":"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3","Type":"ContainerDied","Data":"1d3849088b0aca81ade421007a9b49e8010931bc4e818b694e593426108fa8a2"} Oct 30 18:39:32.749439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.749350 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf" event={"ID":"d5314716-efc8-4ce5-843f-fb4f5f2ba7e3","Type":"ContainerDied","Data":"373ee226b0ce77bd751382b6939f3f87fe1d7dbc92ea737a64f7137dfdce78ca"} Oct 30 18:39:32.749439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.749373 4603 scope.go:122] "RemoveContainer" containerID="1d3849088b0aca81ade421007a9b49e8010931bc4e818b694e593426108fa8a2" Oct 30 18:39:32.760073 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.760057 4603 scope.go:122] "RemoveContainer" containerID="290e03c48d5b4e3be632e3fdaaa0e1f2be7841d7d786197ac96e93474e3cfe88" Oct 30 18:39:32.769385 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.769367 4603 scope.go:122] "RemoveContainer" containerID="1d3849088b0aca81ade421007a9b49e8010931bc4e818b694e593426108fa8a2" Oct 30 18:39:32.769666 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:39:32.769646 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d3849088b0aca81ade421007a9b49e8010931bc4e818b694e593426108fa8a2\": container with ID starting with 1d3849088b0aca81ade421007a9b49e8010931bc4e818b694e593426108fa8a2 not found: ID does not exist" containerID="1d3849088b0aca81ade421007a9b49e8010931bc4e818b694e593426108fa8a2" Oct 30 18:39:32.769722 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.769677 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d3849088b0aca81ade421007a9b49e8010931bc4e818b694e593426108fa8a2"} err="failed to get container status \"1d3849088b0aca81ade421007a9b49e8010931bc4e818b694e593426108fa8a2\": rpc error: code = NotFound desc = could not find container \"1d3849088b0aca81ade421007a9b49e8010931bc4e818b694e593426108fa8a2\": container with ID starting with 1d3849088b0aca81ade421007a9b49e8010931bc4e818b694e593426108fa8a2 not found: ID does not exist" Oct 30 18:39:32.769722 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.769695 4603 scope.go:122] "RemoveContainer" containerID="290e03c48d5b4e3be632e3fdaaa0e1f2be7841d7d786197ac96e93474e3cfe88" Oct 30 18:39:32.769939 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:39:32.769920 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"290e03c48d5b4e3be632e3fdaaa0e1f2be7841d7d786197ac96e93474e3cfe88\": container with ID starting with 290e03c48d5b4e3be632e3fdaaa0e1f2be7841d7d786197ac96e93474e3cfe88 not found: ID does not exist" containerID="290e03c48d5b4e3be632e3fdaaa0e1f2be7841d7d786197ac96e93474e3cfe88" Oct 30 18:39:32.769975 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.769944 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"290e03c48d5b4e3be632e3fdaaa0e1f2be7841d7d786197ac96e93474e3cfe88"} err="failed to get container status \"290e03c48d5b4e3be632e3fdaaa0e1f2be7841d7d786197ac96e93474e3cfe88\": rpc error: code = NotFound desc = could not find container \"290e03c48d5b4e3be632e3fdaaa0e1f2be7841d7d786197ac96e93474e3cfe88\": container with ID starting with 290e03c48d5b4e3be632e3fdaaa0e1f2be7841d7d786197ac96e93474e3cfe88 not found: ID does not exist" Oct 30 18:39:32.816633 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.816607 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf"] Oct 30 18:39:32.828132 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:32.828110 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-58d08b-openshift-default-7b46bbfbd9-2lhmf"] Oct 30 18:39:33.411188 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:33.411157 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" path="/var/lib/kubelet/pods/d5314716-efc8-4ce5-843f-fb4f5f2ba7e3/volumes" Oct 30 18:39:34.093526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:34.089777 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:35.932820 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:35.932611 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:37.459987 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:37.459950 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:37.953145 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:37.953109 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:38.154909 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:38.154860 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:38.356432 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:38.356388 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:38.789165 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:38.789138 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:38.977217 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:38.977181 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:39.658089 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:39.658054 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:39.836947 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:39.836901 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:40.356966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:40.356935 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:40.798340 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:40.797039 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:42.483225 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:42.483148 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:44.200546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:44.196055 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:45.783562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:45.783484 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:46.098049 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:46.097952 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:46.271556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:46.271521 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw"] Oct 30 18:39:46.271832 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:46.271734 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" podUID="0baa3ccf-73e5-4555-9e51-5c627b74124d" containerName="istio-proxy" containerID="cri-o://8f34566063990fb14ef493b7fa4d7f960c042d04c8b9d2d43d0d2c316bcecfec" gracePeriod=30 Oct 30 18:39:46.387678 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:46.386141 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:48.333658 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:48.333552 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:48.874387 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:48.874257 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:49.159735 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:49.159660 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:49.347045 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:49.347011 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np"] Oct 30 18:39:49.347543 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:49.347275 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" podUID="dba942a9-7d94-415a-8491-a17212be1601" containerName="istio-proxy" containerID="cri-o://81f1036c94bcd2e63a7c2329b4824e4d4f21a71c9904da97d891d82f12060fda" gracePeriod=30 Oct 30 18:39:49.587524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:49.586912 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:49.797625 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:49.797579 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:39:49.797921 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:49.797894 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" podUID="0baa3ccf-73e5-4555-9e51-5c627b74124d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:39:51.020937 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.020893 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:51.299326 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.299293 4603 patch_prober.go:29] interesting pod/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:39:51.299457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.299356 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" podUID="dba942a9-7d94-415a-8491-a17212be1601" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:39:51.537151 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.537093 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:39:51.660696 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.660661 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0baa3ccf-73e5-4555-9e51-5c627b74124d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0baa3ccf-73e5-4555-9e51-5c627b74124d-istiod-ca-cert\") pod \"0baa3ccf-73e5-4555-9e51-5c627b74124d\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " Oct 30 18:39:51.660894 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.660739 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-credential-socket\") pod \"0baa3ccf-73e5-4555-9e51-5c627b74124d\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " Oct 30 18:39:51.660894 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.660795 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0baa3ccf-73e5-4555-9e51-5c627b74124d-kube-api-access-jqvxs\" (UniqueName: \"kubernetes.io/projected/0baa3ccf-73e5-4555-9e51-5c627b74124d-kube-api-access-jqvxs\") pod \"0baa3ccf-73e5-4555-9e51-5c627b74124d\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " Oct 30 18:39:51.660894 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.660866 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-token\" (UniqueName: \"kubernetes.io/projected/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-token\") pod \"0baa3ccf-73e5-4555-9e51-5c627b74124d\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " Oct 30 18:39:51.661096 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.660948 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-data\") pod \"0baa3ccf-73e5-4555-9e51-5c627b74124d\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " Oct 30 18:39:51.661096 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.661005 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-envoy\") pod \"0baa3ccf-73e5-4555-9e51-5c627b74124d\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " Oct 30 18:39:51.661096 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.661034 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-podinfo\") pod \"0baa3ccf-73e5-4555-9e51-5c627b74124d\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " Oct 30 18:39:51.661096 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.661072 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-credential-socket" pod "0baa3ccf-73e5-4555-9e51-5c627b74124d" (UID: "0baa3ccf-73e5-4555-9e51-5c627b74124d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:51.661326 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.661121 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-workload-certs\") pod \"0baa3ccf-73e5-4555-9e51-5c627b74124d\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " Oct 30 18:39:51.661326 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.661195 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-workload-socket\") pod \"0baa3ccf-73e5-4555-9e51-5c627b74124d\" (UID: \"0baa3ccf-73e5-4555-9e51-5c627b74124d\") " Oct 30 18:39:51.661326 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.661258 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-data" pod "0baa3ccf-73e5-4555-9e51-5c627b74124d" (UID: "0baa3ccf-73e5-4555-9e51-5c627b74124d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:51.661579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.661542 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-workload-certs" pod "0baa3ccf-73e5-4555-9e51-5c627b74124d" (UID: "0baa3ccf-73e5-4555-9e51-5c627b74124d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:51.661795 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.661772 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0baa3ccf-73e5-4555-9e51-5c627b74124d-istiod-ca-cert" pod "0baa3ccf-73e5-4555-9e51-5c627b74124d" (UID: "0baa3ccf-73e5-4555-9e51-5c627b74124d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:51.662295 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.662058 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.662455 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.662429 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.662597 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.662457 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.662597 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.662554 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-workload-socket" pod "0baa3ccf-73e5-4555-9e51-5c627b74124d" (UID: "0baa3ccf-73e5-4555-9e51-5c627b74124d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:51.663556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.663489 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-envoy" pod "0baa3ccf-73e5-4555-9e51-5c627b74124d" (UID: "0baa3ccf-73e5-4555-9e51-5c627b74124d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:51.663739 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.663719 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-podinfo" pod "0baa3ccf-73e5-4555-9e51-5c627b74124d" (UID: "0baa3ccf-73e5-4555-9e51-5c627b74124d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:39:51.663895 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.663878 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0baa3ccf-73e5-4555-9e51-5c627b74124d-kube-api-access-jqvxs" pod "0baa3ccf-73e5-4555-9e51-5c627b74124d" (UID: "0baa3ccf-73e5-4555-9e51-5c627b74124d"). InnerVolumeSpecName "kube-api-access-jqvxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:51.664137 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.664117 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-token" pod "0baa3ccf-73e5-4555-9e51-5c627b74124d" (UID: "0baa3ccf-73e5-4555-9e51-5c627b74124d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:51.763904 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.763873 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.763904 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.763902 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.763904 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.763911 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baa3ccf-73e5-4555-9e51-5c627b74124d-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.764190 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.763920 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0baa3ccf-73e5-4555-9e51-5c627b74124d-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.764190 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.763930 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqvxs\" (UniqueName: \"kubernetes.io/projected/0baa3ccf-73e5-4555-9e51-5c627b74124d-kube-api-access-jqvxs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.764190 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.763938 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0baa3ccf-73e5-4555-9e51-5c627b74124d-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.930911 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:39:51.930824 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:39:51.990528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.990466 4603 generic.go:356] "Generic (PLEG): container finished" podID="0baa3ccf-73e5-4555-9e51-5c627b74124d" containerID="8f34566063990fb14ef493b7fa4d7f960c042d04c8b9d2d43d0d2c316bcecfec" exitCode=0 Oct 30 18:39:51.990704 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.990588 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" Oct 30 18:39:51.990704 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.990617 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" event={"ID":"0baa3ccf-73e5-4555-9e51-5c627b74124d","Type":"ContainerDied","Data":"8f34566063990fb14ef493b7fa4d7f960c042d04c8b9d2d43d0d2c316bcecfec"} Oct 30 18:39:51.990704 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.990681 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw" event={"ID":"0baa3ccf-73e5-4555-9e51-5c627b74124d","Type":"ContainerDied","Data":"574511a27d2bf96a841fd2b1584a55dfcc90fed775d02f215f6e54093eb987a4"} Oct 30 18:39:51.990830 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:51.990713 4603 scope.go:122] "RemoveContainer" containerID="8f34566063990fb14ef493b7fa4d7f960c042d04c8b9d2d43d0d2c316bcecfec" Oct 30 18:39:52.002405 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:52.002384 4603 scope.go:122] "RemoveContainer" containerID="e627332c30048dc62c7abd519e19bafc165413f069b36427fb4456800e4971e9" Oct 30 18:39:52.012011 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:52.011993 4603 scope.go:122] "RemoveContainer" containerID="8f34566063990fb14ef493b7fa4d7f960c042d04c8b9d2d43d0d2c316bcecfec" Oct 30 18:39:52.012292 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:39:52.012273 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f34566063990fb14ef493b7fa4d7f960c042d04c8b9d2d43d0d2c316bcecfec\": container with ID starting with 8f34566063990fb14ef493b7fa4d7f960c042d04c8b9d2d43d0d2c316bcecfec not found: ID does not exist" containerID="8f34566063990fb14ef493b7fa4d7f960c042d04c8b9d2d43d0d2c316bcecfec" Oct 30 18:39:52.012377 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:52.012303 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f34566063990fb14ef493b7fa4d7f960c042d04c8b9d2d43d0d2c316bcecfec"} err="failed to get container status \"8f34566063990fb14ef493b7fa4d7f960c042d04c8b9d2d43d0d2c316bcecfec\": rpc error: code = NotFound desc = could not find container \"8f34566063990fb14ef493b7fa4d7f960c042d04c8b9d2d43d0d2c316bcecfec\": container with ID starting with 8f34566063990fb14ef493b7fa4d7f960c042d04c8b9d2d43d0d2c316bcecfec not found: ID does not exist" Oct 30 18:39:52.012377 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:52.012321 4603 scope.go:122] "RemoveContainer" containerID="e627332c30048dc62c7abd519e19bafc165413f069b36427fb4456800e4971e9" Oct 30 18:39:52.012619 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:39:52.012603 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e627332c30048dc62c7abd519e19bafc165413f069b36427fb4456800e4971e9\": container with ID starting with e627332c30048dc62c7abd519e19bafc165413f069b36427fb4456800e4971e9 not found: ID does not exist" containerID="e627332c30048dc62c7abd519e19bafc165413f069b36427fb4456800e4971e9" Oct 30 18:39:52.012686 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:52.012624 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e627332c30048dc62c7abd519e19bafc165413f069b36427fb4456800e4971e9"} err="failed to get container status \"e627332c30048dc62c7abd519e19bafc165413f069b36427fb4456800e4971e9\": rpc error: code = NotFound desc = could not find container \"e627332c30048dc62c7abd519e19bafc165413f069b36427fb4456800e4971e9\": container with ID starting with e627332c30048dc62c7abd519e19bafc165413f069b36427fb4456800e4971e9 not found: ID does not exist" Oct 30 18:39:52.048919 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:52.048890 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw"] Oct 30 18:39:52.060661 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:52.060638 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-6k5jw"] Oct 30 18:39:53.370230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:53.370199 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:53.412967 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:53.412929 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0baa3ccf-73e5-4555-9e51-5c627b74124d" path="/var/lib/kubelet/pods/0baa3ccf-73e5-4555-9e51-5c627b74124d/volumes" Oct 30 18:39:54.632369 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.632347 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:39:54.700030 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.699998 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dba942a9-7d94-415a-8491-a17212be1601-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dba942a9-7d94-415a-8491-a17212be1601-istio-podinfo\") pod \"dba942a9-7d94-415a-8491-a17212be1601\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " Oct 30 18:39:54.700229 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.700061 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-workload-certs\") pod \"dba942a9-7d94-415a-8491-a17212be1601\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " Oct 30 18:39:54.700229 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.700098 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-workload-socket\") pod \"dba942a9-7d94-415a-8491-a17212be1601\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " Oct 30 18:39:54.700229 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.700197 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-credential-socket\") pod \"dba942a9-7d94-415a-8491-a17212be1601\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " Oct 30 18:39:54.700423 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.700252 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-istio-envoy\") pod \"dba942a9-7d94-415a-8491-a17212be1601\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " Oct 30 18:39:54.700423 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.700313 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-istio-data\") pod \"dba942a9-7d94-415a-8491-a17212be1601\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " Oct 30 18:39:54.700423 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.700371 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dba942a9-7d94-415a-8491-a17212be1601-istio-token\" (UniqueName: \"kubernetes.io/projected/dba942a9-7d94-415a-8491-a17212be1601-istio-token\") pod \"dba942a9-7d94-415a-8491-a17212be1601\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " Oct 30 18:39:54.700637 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.700431 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dba942a9-7d94-415a-8491-a17212be1601-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dba942a9-7d94-415a-8491-a17212be1601-istiod-ca-cert\") pod \"dba942a9-7d94-415a-8491-a17212be1601\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " Oct 30 18:39:54.700637 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.700432 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-workload-certs" pod "dba942a9-7d94-415a-8491-a17212be1601" (UID: "dba942a9-7d94-415a-8491-a17212be1601"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:54.700637 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.700478 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dba942a9-7d94-415a-8491-a17212be1601-kube-api-access-nrs9n\" (UniqueName: \"kubernetes.io/projected/dba942a9-7d94-415a-8491-a17212be1601-kube-api-access-nrs9n\") pod \"dba942a9-7d94-415a-8491-a17212be1601\" (UID: \"dba942a9-7d94-415a-8491-a17212be1601\") " Oct 30 18:39:54.700637 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.700584 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-credential-socket" pod "dba942a9-7d94-415a-8491-a17212be1601" (UID: "dba942a9-7d94-415a-8491-a17212be1601"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:54.701033 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.700704 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-istio-data" pod "dba942a9-7d94-415a-8491-a17212be1601" (UID: "dba942a9-7d94-415a-8491-a17212be1601"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:54.701097 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.701032 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-workload-socket" pod "dba942a9-7d94-415a-8491-a17212be1601" (UID: "dba942a9-7d94-415a-8491-a17212be1601"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:54.701191 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.701168 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:54.701261 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.701196 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:54.701261 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.701211 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:54.701261 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.701226 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:54.701440 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.701268 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba942a9-7d94-415a-8491-a17212be1601-istiod-ca-cert" pod "dba942a9-7d94-415a-8491-a17212be1601" (UID: "dba942a9-7d94-415a-8491-a17212be1601"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:54.702647 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.702626 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dba942a9-7d94-415a-8491-a17212be1601-istio-podinfo" pod "dba942a9-7d94-415a-8491-a17212be1601" (UID: "dba942a9-7d94-415a-8491-a17212be1601"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:39:54.703555 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.703535 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba942a9-7d94-415a-8491-a17212be1601-istio-token" pod "dba942a9-7d94-415a-8491-a17212be1601" (UID: "dba942a9-7d94-415a-8491-a17212be1601"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:54.703742 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.703715 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba942a9-7d94-415a-8491-a17212be1601-kube-api-access-nrs9n" pod "dba942a9-7d94-415a-8491-a17212be1601" (UID: "dba942a9-7d94-415a-8491-a17212be1601"). InnerVolumeSpecName "kube-api-access-nrs9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:54.703742 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.703719 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-istio-envoy" pod "dba942a9-7d94-415a-8491-a17212be1601" (UID: "dba942a9-7d94-415a-8491-a17212be1601"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:54.802451 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.802419 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dba942a9-7d94-415a-8491-a17212be1601-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:54.802451 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.802447 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dba942a9-7d94-415a-8491-a17212be1601-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:54.802451 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.802455 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dba942a9-7d94-415a-8491-a17212be1601-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:54.802704 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.802465 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrs9n\" (UniqueName: \"kubernetes.io/projected/dba942a9-7d94-415a-8491-a17212be1601-kube-api-access-nrs9n\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:54.802704 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:54.802476 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dba942a9-7d94-415a-8491-a17212be1601-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:39:55.012423 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:55.012339 4603 generic.go:356] "Generic (PLEG): container finished" podID="dba942a9-7d94-415a-8491-a17212be1601" containerID="81f1036c94bcd2e63a7c2329b4824e4d4f21a71c9904da97d891d82f12060fda" exitCode=0 Oct 30 18:39:55.012423 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:55.012397 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" Oct 30 18:39:55.012632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:55.012432 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" event={"ID":"dba942a9-7d94-415a-8491-a17212be1601","Type":"ContainerDied","Data":"81f1036c94bcd2e63a7c2329b4824e4d4f21a71c9904da97d891d82f12060fda"} Oct 30 18:39:55.012632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:55.012540 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np" event={"ID":"dba942a9-7d94-415a-8491-a17212be1601","Type":"ContainerDied","Data":"fded3bb65a2a83c53ec2a723501073b82a627b4836c5dbc954f0974a1e973df7"} Oct 30 18:39:55.012632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:55.012571 4603 scope.go:122] "RemoveContainer" containerID="81f1036c94bcd2e63a7c2329b4824e4d4f21a71c9904da97d891d82f12060fda" Oct 30 18:39:55.023859 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:55.023842 4603 scope.go:122] "RemoveContainer" containerID="38a01e2bc6fb4ba12131c55003733c74e3fff4f17fb808240979c542a9be32b4" Oct 30 18:39:55.032489 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:55.032474 4603 scope.go:122] "RemoveContainer" containerID="81f1036c94bcd2e63a7c2329b4824e4d4f21a71c9904da97d891d82f12060fda" Oct 30 18:39:55.032788 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:39:55.032768 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f1036c94bcd2e63a7c2329b4824e4d4f21a71c9904da97d891d82f12060fda\": container with ID starting with 81f1036c94bcd2e63a7c2329b4824e4d4f21a71c9904da97d891d82f12060fda not found: ID does not exist" containerID="81f1036c94bcd2e63a7c2329b4824e4d4f21a71c9904da97d891d82f12060fda" Oct 30 18:39:55.032867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:55.032808 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f1036c94bcd2e63a7c2329b4824e4d4f21a71c9904da97d891d82f12060fda"} err="failed to get container status \"81f1036c94bcd2e63a7c2329b4824e4d4f21a71c9904da97d891d82f12060fda\": rpc error: code = NotFound desc = could not find container \"81f1036c94bcd2e63a7c2329b4824e4d4f21a71c9904da97d891d82f12060fda\": container with ID starting with 81f1036c94bcd2e63a7c2329b4824e4d4f21a71c9904da97d891d82f12060fda not found: ID does not exist" Oct 30 18:39:55.032867 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:55.032825 4603 scope.go:122] "RemoveContainer" containerID="38a01e2bc6fb4ba12131c55003733c74e3fff4f17fb808240979c542a9be32b4" Oct 30 18:39:55.033089 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:39:55.033071 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38a01e2bc6fb4ba12131c55003733c74e3fff4f17fb808240979c542a9be32b4\": container with ID starting with 38a01e2bc6fb4ba12131c55003733c74e3fff4f17fb808240979c542a9be32b4 not found: ID does not exist" containerID="38a01e2bc6fb4ba12131c55003733c74e3fff4f17fb808240979c542a9be32b4" Oct 30 18:39:55.033136 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:55.033097 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38a01e2bc6fb4ba12131c55003733c74e3fff4f17fb808240979c542a9be32b4"} err="failed to get container status \"38a01e2bc6fb4ba12131c55003733c74e3fff4f17fb808240979c542a9be32b4\": rpc error: code = NotFound desc = could not find container \"38a01e2bc6fb4ba12131c55003733c74e3fff4f17fb808240979c542a9be32b4\": container with ID starting with 38a01e2bc6fb4ba12131c55003733c74e3fff4f17fb808240979c542a9be32b4 not found: ID does not exist" Oct 30 18:39:55.064792 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:55.064762 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np"] Oct 30 18:39:55.076579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:55.076556 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-50e9fc-openshift-default-cb4d69df5-5g5np"] Oct 30 18:39:55.190603 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:55.188698 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:55.410294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:55.410265 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dba942a9-7d94-415a-8491-a17212be1601" path="/var/lib/kubelet/pods/dba942a9-7d94-415a-8491-a17212be1601/volumes" Oct 30 18:39:56.908529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:56.907818 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:57.431206 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:57.431164 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc"] Oct 30 18:39:57.478138 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:57.478097 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp"] Oct 30 18:39:57.653681 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:57.653647 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:57.846838 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:57.846809 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:58.669557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:58.669416 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:59.770689 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:39:59.770652 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:40:00.384342 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:00.384245 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:00.443611 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:00.443581 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j"] Oct 30 18:40:00.480658 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:00.480626 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw"] Oct 30 18:40:00.655966 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:00.655861 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:00.803087 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:00.803055 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:01.951948 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:01.951910 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:02.452663 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:02.452585 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc" podUID="c41e57c9-3c80-40fc-8ab4-8b3df57b0136" containerName="payload-processing" containerID="cri-o://6fd54d6326384b91bfb34d85ecee4d31bef6d3b31056594675c175ef7512edf9" gracePeriod=25 Oct 30 18:40:02.478546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:02.478466 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp" podUID="d99bf6c9-a08f-4f49-b845-0f411f1f1720" containerName="payload-pre-processing" containerID="cri-o://34b3f2d7c8d7ee0152a6cb46bf226ddef3f0388233e6f727e8a9a694816c812d" gracePeriod=25 Oct 30 18:40:03.287032 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:03.286928 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:03.315801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:03.314410 4603 generic.go:356] "Generic (PLEG): container finished" podID="c41e57c9-3c80-40fc-8ab4-8b3df57b0136" containerID="6fd54d6326384b91bfb34d85ecee4d31bef6d3b31056594675c175ef7512edf9" exitCode=0 Oct 30 18:40:03.315801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:03.314662 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc" event={"ID":"c41e57c9-3c80-40fc-8ab4-8b3df57b0136","Type":"ContainerDied","Data":"6fd54d6326384b91bfb34d85ecee4d31bef6d3b31056594675c175ef7512edf9"} Oct 30 18:40:03.329746 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:03.329715 4603 generic.go:356] "Generic (PLEG): container finished" podID="d99bf6c9-a08f-4f49-b845-0f411f1f1720" containerID="34b3f2d7c8d7ee0152a6cb46bf226ddef3f0388233e6f727e8a9a694816c812d" exitCode=0 Oct 30 18:40:03.330114 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:03.330086 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp" event={"ID":"d99bf6c9-a08f-4f49-b845-0f411f1f1720","Type":"ContainerDied","Data":"34b3f2d7c8d7ee0152a6cb46bf226ddef3f0388233e6f727e8a9a694816c812d"} Oct 30 18:40:03.475046 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:03.470727 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:03.614228 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:03.612593 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:03.641550 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:03.638258 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:03.727554 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:03.717798 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc" Oct 30 18:40:03.764747 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:03.764712 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c41e57c9-3c80-40fc-8ab4-8b3df57b0136-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c41e57c9-3c80-40fc-8ab4-8b3df57b0136-plugins-config-volume\") pod \"c41e57c9-3c80-40fc-8ab4-8b3df57b0136\" (UID: \"c41e57c9-3c80-40fc-8ab4-8b3df57b0136\") " Oct 30 18:40:03.765768 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:03.765743 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c41e57c9-3c80-40fc-8ab4-8b3df57b0136-kube-api-access-tjxn6\" (UniqueName: \"kubernetes.io/projected/c41e57c9-3c80-40fc-8ab4-8b3df57b0136-kube-api-access-tjxn6\") pod \"c41e57c9-3c80-40fc-8ab4-8b3df57b0136\" (UID: \"c41e57c9-3c80-40fc-8ab4-8b3df57b0136\") " Oct 30 18:40:03.766360 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:03.765371 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c41e57c9-3c80-40fc-8ab4-8b3df57b0136-plugins-config-volume" pod "c41e57c9-3c80-40fc-8ab4-8b3df57b0136" (UID: "c41e57c9-3c80-40fc-8ab4-8b3df57b0136"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:03.810395 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:03.766988 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c41e57c9-3c80-40fc-8ab4-8b3df57b0136-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:03.873588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:03.870810 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:03.973535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:03.943825 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c41e57c9-3c80-40fc-8ab4-8b3df57b0136-kube-api-access-tjxn6" pod "c41e57c9-3c80-40fc-8ab4-8b3df57b0136" (UID: "c41e57c9-3c80-40fc-8ab4-8b3df57b0136"). InnerVolumeSpecName "kube-api-access-tjxn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:04.036526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.032574 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjxn6\" (UniqueName: \"kubernetes.io/projected/c41e57c9-3c80-40fc-8ab4-8b3df57b0136-kube-api-access-tjxn6\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:04.036526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.033698 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp" Oct 30 18:40:04.139586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.139014 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d99bf6c9-a08f-4f49-b845-0f411f1f1720-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d99bf6c9-a08f-4f49-b845-0f411f1f1720-plugins-config-volume\") pod \"d99bf6c9-a08f-4f49-b845-0f411f1f1720\" (UID: \"d99bf6c9-a08f-4f49-b845-0f411f1f1720\") " Oct 30 18:40:04.139586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.139111 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d99bf6c9-a08f-4f49-b845-0f411f1f1720-kube-api-access-tfds6\" (UniqueName: \"kubernetes.io/projected/d99bf6c9-a08f-4f49-b845-0f411f1f1720-kube-api-access-tfds6\") pod \"d99bf6c9-a08f-4f49-b845-0f411f1f1720\" (UID: \"d99bf6c9-a08f-4f49-b845-0f411f1f1720\") " Oct 30 18:40:04.143537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.141715 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d99bf6c9-a08f-4f49-b845-0f411f1f1720-plugins-config-volume" pod "d99bf6c9-a08f-4f49-b845-0f411f1f1720" (UID: "d99bf6c9-a08f-4f49-b845-0f411f1f1720"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:04.170133 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.170069 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d99bf6c9-a08f-4f49-b845-0f411f1f1720-kube-api-access-tfds6" pod "d99bf6c9-a08f-4f49-b845-0f411f1f1720" (UID: "d99bf6c9-a08f-4f49-b845-0f411f1f1720"). InnerVolumeSpecName "kube-api-access-tfds6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:04.262527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.254976 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:40:04.266524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.266313 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d99bf6c9-a08f-4f49-b845-0f411f1f1720-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:04.266524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.266348 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfds6\" (UniqueName: \"kubernetes.io/projected/d99bf6c9-a08f-4f49-b845-0f411f1f1720-kube-api-access-tfds6\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:04.311531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.263880 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" podUID="81d31061-bdab-430c-9dba-d50879b97a29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.176:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:04.445584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.444672 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:04.449526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.447705 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:04.478533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.477952 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc" event={"ID":"c41e57c9-3c80-40fc-8ab4-8b3df57b0136","Type":"ContainerDied","Data":"4b56ce2969b8fea8f285ece31afc96ee5a40dd190a3a25748e056ef2bd6fff71"} Oct 30 18:40:04.478533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.478028 4603 scope.go:122] "RemoveContainer" containerID="6fd54d6326384b91bfb34d85ecee4d31bef6d3b31056594675c175ef7512edf9" Oct 30 18:40:04.478533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.478201 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc" Oct 30 18:40:04.513344 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.513312 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp" event={"ID":"d99bf6c9-a08f-4f49-b845-0f411f1f1720","Type":"ContainerDied","Data":"e446a93412ab31ec14a8850fb008a97135b956ba0e85986e6ce8d714866b44db"} Oct 30 18:40:04.513735 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.513653 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp" Oct 30 18:40:04.569200 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.569155 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:40:04.604270 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.604233 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Oct 30 18:40:04.611551 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.609995 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:04.614548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.613582 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:04.614548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.614092 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:40:04.665894 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.665866 4603 scope.go:122] "RemoveContainer" containerID="34b3f2d7c8d7ee0152a6cb46bf226ddef3f0388233e6f727e8a9a694816c812d" Oct 30 18:40:04.736587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.711748 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc"] Oct 30 18:40:04.793045 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.789047 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c8eef3-59f868587b-vc6dc"] Oct 30 18:40:04.803419 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.803382 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp"] Oct 30 18:40:04.803698 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:04.803678 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c8eef3-6d879b4f7f-vdtzp"] Oct 30 18:40:05.261536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:05.259597 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:05.261536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:05.260018 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:05.261536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:05.260799 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:40:05.304837 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:05.303957 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" podUID="81d31061-bdab-430c-9dba-d50879b97a29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.176:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:05.304837 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:05.304820 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:40:05.331203 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:05.328947 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:40:05.462733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:05.460652 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:05.462733 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:05.461884 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j" podUID="70342038-ca6c-4a23-980b-7b750af92367" containerName="payload-processing" containerID="cri-o://9072d87870a27eb2725933e9f9637ec2be47b30e1812b9dbfab97a3778a0e708" gracePeriod=25 Oct 30 18:40:05.485598 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:05.485556 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw" podUID="9e2e8d0b-1be6-4e07-aa98-db2472756ee5" containerName="payload-pre-processing" containerID="cri-o://b34a6cf8ba2569641d9186e13fc4c33879c0f5616f0df2180949f1ca697b75fe" gracePeriod=25 Oct 30 18:40:05.640911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:05.510229 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:05.640911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:05.609451 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:05.795559 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:05.774255 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c41e57c9-3c80-40fc-8ab4-8b3df57b0136" path="/var/lib/kubelet/pods/c41e57c9-3c80-40fc-8ab4-8b3df57b0136/volumes" Oct 30 18:40:05.796461 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:05.796431 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d99bf6c9-a08f-4f49-b845-0f411f1f1720" path="/var/lib/kubelet/pods/d99bf6c9-a08f-4f49-b845-0f411f1f1720/volumes" Oct 30 18:40:05.797153 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:05.797131 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:40:05.797330 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:05.797315 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:40:06.298535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:06.291898 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:06.298535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:06.295137 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:40:06.298535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:06.296661 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:40:06.331544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:06.329799 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" podUID="81d31061-bdab-430c-9dba-d50879b97a29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.176:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:06.331544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:06.331031 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:06.571921 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:06.571877 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:40:06.600385 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:06.600341 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:40:06.633762 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:06.633714 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:06.650524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:06.647352 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:06.789610 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:06.789561 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:40:06.827302 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:06.827218 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:07.228964 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:07.228632 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:07.308762 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:07.308712 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:40:07.335530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:07.333648 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:07.335530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:07.334054 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" podUID="81d31061-bdab-430c-9dba-d50879b97a29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.176:8000/ready\": context deadline exceeded" Oct 30 18:40:07.335530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:07.334807 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:40:07.385186 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:07.385132 4603 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded" Oct 30 18:40:07.634343 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:07.634300 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 18:40:07.651442 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:07.651013 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:07.756371 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:07.744627 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:07.792834 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:07.792162 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:07.829547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:07.828733 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:40:07.829547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:07.828855 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:40:07.829547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:07.828893 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:40:07.829547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:07.828924 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:40:07.829547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:07.828951 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:40:07.953928 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:07.945556 4603 generic.go:356] "Generic (PLEG): container finished" podID="70342038-ca6c-4a23-980b-7b750af92367" containerID="9072d87870a27eb2725933e9f9637ec2be47b30e1812b9dbfab97a3778a0e708" exitCode=0 Oct 30 18:40:07.953928 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:07.945796 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j" event={"ID":"70342038-ca6c-4a23-980b-7b750af92367","Type":"ContainerDied","Data":"9072d87870a27eb2725933e9f9637ec2be47b30e1812b9dbfab97a3778a0e708"} Oct 30 18:40:07.953928 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:07.948175 4603 generic.go:356] "Generic (PLEG): container finished" podID="9e2e8d0b-1be6-4e07-aa98-db2472756ee5" containerID="b34a6cf8ba2569641d9186e13fc4c33879c0f5616f0df2180949f1ca697b75fe" exitCode=0 Oct 30 18:40:07.953928 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:07.949169 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw" event={"ID":"9e2e8d0b-1be6-4e07-aa98-db2472756ee5","Type":"ContainerDied","Data":"b34a6cf8ba2569641d9186e13fc4c33879c0f5616f0df2180949f1ca697b75fe"} Oct 30 18:40:07.966916 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:07.966830 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:40:07.974346 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:07.974324 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" Oct 30 18:40:07.975164 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:07.975144 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:40:07.984523 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:07.984474 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:40:08.073575 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:08.072573 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw" Oct 30 18:40:08.162871 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:08.162808 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j" Oct 30 18:40:08.251047 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:08.251020 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70342038-ca6c-4a23-980b-7b750af92367-kube-api-access-hphf7\" (UniqueName: \"kubernetes.io/projected/70342038-ca6c-4a23-980b-7b750af92367-kube-api-access-hphf7\") pod \"70342038-ca6c-4a23-980b-7b750af92367\" (UID: \"70342038-ca6c-4a23-980b-7b750af92367\") " Oct 30 18:40:08.251222 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:08.251092 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e2e8d0b-1be6-4e07-aa98-db2472756ee5-kube-api-access-crtx2\" (UniqueName: \"kubernetes.io/projected/9e2e8d0b-1be6-4e07-aa98-db2472756ee5-kube-api-access-crtx2\") pod \"9e2e8d0b-1be6-4e07-aa98-db2472756ee5\" (UID: \"9e2e8d0b-1be6-4e07-aa98-db2472756ee5\") " Oct 30 18:40:08.251222 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:08.251187 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70342038-ca6c-4a23-980b-7b750af92367-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70342038-ca6c-4a23-980b-7b750af92367-plugins-config-volume\") pod \"70342038-ca6c-4a23-980b-7b750af92367\" (UID: \"70342038-ca6c-4a23-980b-7b750af92367\") " Oct 30 18:40:08.251306 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:08.251250 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e2e8d0b-1be6-4e07-aa98-db2472756ee5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e2e8d0b-1be6-4e07-aa98-db2472756ee5-plugins-config-volume\") pod \"9e2e8d0b-1be6-4e07-aa98-db2472756ee5\" (UID: \"9e2e8d0b-1be6-4e07-aa98-db2472756ee5\") " Oct 30 18:40:08.251651 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:08.251624 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70342038-ca6c-4a23-980b-7b750af92367-plugins-config-volume" pod "70342038-ca6c-4a23-980b-7b750af92367" (UID: "70342038-ca6c-4a23-980b-7b750af92367"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:08.251962 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:08.251661 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e2e8d0b-1be6-4e07-aa98-db2472756ee5-plugins-config-volume" pod "9e2e8d0b-1be6-4e07-aa98-db2472756ee5" (UID: "9e2e8d0b-1be6-4e07-aa98-db2472756ee5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:08.252042 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:08.251993 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70342038-ca6c-4a23-980b-7b750af92367-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:08.252042 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:08.252013 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e2e8d0b-1be6-4e07-aa98-db2472756ee5-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:08.253636 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:08.253609 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2e8d0b-1be6-4e07-aa98-db2472756ee5-kube-api-access-crtx2" pod "9e2e8d0b-1be6-4e07-aa98-db2472756ee5" (UID: "9e2e8d0b-1be6-4e07-aa98-db2472756ee5"). InnerVolumeSpecName "kube-api-access-crtx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:08.253803 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:08.253669 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70342038-ca6c-4a23-980b-7b750af92367-kube-api-access-hphf7" pod "70342038-ca6c-4a23-980b-7b750af92367" (UID: "70342038-ca6c-4a23-980b-7b750af92367"). InnerVolumeSpecName "kube-api-access-hphf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:08.279147 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:08.279117 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:08.353863 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:08.353827 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hphf7\" (UniqueName: \"kubernetes.io/projected/70342038-ca6c-4a23-980b-7b750af92367-kube-api-access-hphf7\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:08.353863 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:08.353865 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crtx2\" (UniqueName: \"kubernetes.io/projected/9e2e8d0b-1be6-4e07-aa98-db2472756ee5-kube-api-access-crtx2\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:08.933715 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:08.933674 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:08.955261 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:08.955236 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw" Oct 30 18:40:08.955450 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:08.955272 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw" event={"ID":"9e2e8d0b-1be6-4e07-aa98-db2472756ee5","Type":"ContainerDied","Data":"c4476e4a5db72bd3e1e2207536328e0ff77ef649b01888adbb45aeeb9237dce4"} Oct 30 18:40:08.955450 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:08.955335 4603 scope.go:122] "RemoveContainer" containerID="b34a6cf8ba2569641d9186e13fc4c33879c0f5616f0df2180949f1ca697b75fe" Oct 30 18:40:08.956833 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:08.956814 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j" Oct 30 18:40:08.956833 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:08.956826 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j" event={"ID":"70342038-ca6c-4a23-980b-7b750af92367","Type":"ContainerDied","Data":"6ada50fc79e68b8149b35dd4d51671c9ea50aed6646cd1cded9a5476e482fe84"} Oct 30 18:40:08.966518 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:08.966317 4603 scope.go:122] "RemoveContainer" containerID="9072d87870a27eb2725933e9f9637ec2be47b30e1812b9dbfab97a3778a0e708" Oct 30 18:40:09.009567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:09.009544 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j"] Oct 30 18:40:09.021488 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:09.021466 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-3b8bf3-6d99f66f54-cv67j"] Oct 30 18:40:09.057494 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:09.057464 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw"] Oct 30 18:40:09.062679 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:09.062658 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-3b8bf3-644c5dcf95-qpfcw"] Oct 30 18:40:09.122267 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:09.122241 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:09.414662 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:09.414621 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70342038-ca6c-4a23-980b-7b750af92367" path="/var/lib/kubelet/pods/70342038-ca6c-4a23-980b-7b750af92367/volumes" Oct 30 18:40:09.415439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:09.415409 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e2e8d0b-1be6-4e07-aa98-db2472756ee5" path="/var/lib/kubelet/pods/9e2e8d0b-1be6-4e07-aa98-db2472756ee5/volumes" Oct 30 18:40:09.671987 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:09.671906 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:10.879880 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:10.879841 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:11.048597 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:11.048560 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:11.225326 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:11.225178 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:11.233000 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:11.232974 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt"] Oct 30 18:40:11.233204 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:11.233180 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" podUID="f9bcc5bf-0143-429f-95c1-298df70d5f88" containerName="istio-proxy" containerID="cri-o://dd9711d5b66007469ce86d91263f92ac85c307a18897fff91c3361931f121022" gracePeriod=30 Oct 30 18:40:11.393520 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:11.393415 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:11.823286 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:11.823244 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:13.057147 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:13.057099 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:14.806602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:14.806478 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:16.490754 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.490730 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:40:16.577125 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.577094 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9bcc5bf-0143-429f-95c1-298df70d5f88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bcc5bf-0143-429f-95c1-298df70d5f88-istiod-ca-cert\") pod \"f9bcc5bf-0143-429f-95c1-298df70d5f88\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " Oct 30 18:40:16.577314 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.577205 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-token\" (UniqueName: \"kubernetes.io/projected/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-token\") pod \"f9bcc5bf-0143-429f-95c1-298df70d5f88\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " Oct 30 18:40:16.577314 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.577246 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-envoy\") pod \"f9bcc5bf-0143-429f-95c1-298df70d5f88\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " Oct 30 18:40:16.577457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.577357 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-data\") pod \"f9bcc5bf-0143-429f-95c1-298df70d5f88\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " Oct 30 18:40:16.577457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.577399 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9bcc5bf-0143-429f-95c1-298df70d5f88-kube-api-access-6c88h\" (UniqueName: \"kubernetes.io/projected/f9bcc5bf-0143-429f-95c1-298df70d5f88-kube-api-access-6c88h\") pod \"f9bcc5bf-0143-429f-95c1-298df70d5f88\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " Oct 30 18:40:16.577457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.577450 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-credential-socket\") pod \"f9bcc5bf-0143-429f-95c1-298df70d5f88\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " Oct 30 18:40:16.577654 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.577493 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-workload-socket\") pod \"f9bcc5bf-0143-429f-95c1-298df70d5f88\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " Oct 30 18:40:16.577654 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.577576 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-podinfo\") pod \"f9bcc5bf-0143-429f-95c1-298df70d5f88\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " Oct 30 18:40:16.577654 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.577632 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-workload-certs\") pod \"f9bcc5bf-0143-429f-95c1-298df70d5f88\" (UID: \"f9bcc5bf-0143-429f-95c1-298df70d5f88\") " Oct 30 18:40:16.577906 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.577881 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9bcc5bf-0143-429f-95c1-298df70d5f88-istiod-ca-cert" pod "f9bcc5bf-0143-429f-95c1-298df70d5f88" (UID: "f9bcc5bf-0143-429f-95c1-298df70d5f88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:16.578304 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.578279 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-workload-certs" pod "f9bcc5bf-0143-429f-95c1-298df70d5f88" (UID: "f9bcc5bf-0143-429f-95c1-298df70d5f88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:16.578387 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.578367 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bcc5bf-0143-429f-95c1-298df70d5f88-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:16.578933 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.578605 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-credential-socket" pod "f9bcc5bf-0143-429f-95c1-298df70d5f88" (UID: "f9bcc5bf-0143-429f-95c1-298df70d5f88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:16.578933 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.578798 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-data" pod "f9bcc5bf-0143-429f-95c1-298df70d5f88" (UID: "f9bcc5bf-0143-429f-95c1-298df70d5f88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:16.579469 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.579438 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-workload-socket" pod "f9bcc5bf-0143-429f-95c1-298df70d5f88" (UID: "f9bcc5bf-0143-429f-95c1-298df70d5f88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:16.580928 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.580902 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-envoy" pod "f9bcc5bf-0143-429f-95c1-298df70d5f88" (UID: "f9bcc5bf-0143-429f-95c1-298df70d5f88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:16.581119 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.581094 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-token" pod "f9bcc5bf-0143-429f-95c1-298df70d5f88" (UID: "f9bcc5bf-0143-429f-95c1-298df70d5f88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:16.581544 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.581519 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bcc5bf-0143-429f-95c1-298df70d5f88-kube-api-access-6c88h" pod "f9bcc5bf-0143-429f-95c1-298df70d5f88" (UID: "f9bcc5bf-0143-429f-95c1-298df70d5f88"). InnerVolumeSpecName "kube-api-access-6c88h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:16.581779 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.581754 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-podinfo" pod "f9bcc5bf-0143-429f-95c1-298df70d5f88" (UID: "f9bcc5bf-0143-429f-95c1-298df70d5f88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:40:16.655643 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.655614 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:16.698701 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.698657 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:16.698701 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.698701 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c88h\" (UniqueName: \"kubernetes.io/projected/f9bcc5bf-0143-429f-95c1-298df70d5f88-kube-api-access-6c88h\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:16.699010 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.698718 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:16.699010 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.698729 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:16.699010 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.698739 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:16.699010 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.698747 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:16.699010 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.698758 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:16.699010 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.698765 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bcc5bf-0143-429f-95c1-298df70d5f88-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:16.865730 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.865640 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2"] Oct 30 18:40:16.888084 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:16.888058 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z"] Oct 30 18:40:17.008873 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:17.008840 4603 generic.go:356] "Generic (PLEG): container finished" podID="f9bcc5bf-0143-429f-95c1-298df70d5f88" containerID="dd9711d5b66007469ce86d91263f92ac85c307a18897fff91c3361931f121022" exitCode=0 Oct 30 18:40:17.008873 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:17.008874 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" Oct 30 18:40:17.009119 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:17.008896 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" event={"ID":"f9bcc5bf-0143-429f-95c1-298df70d5f88","Type":"ContainerDied","Data":"dd9711d5b66007469ce86d91263f92ac85c307a18897fff91c3361931f121022"} Oct 30 18:40:17.009119 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:17.008952 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt" event={"ID":"f9bcc5bf-0143-429f-95c1-298df70d5f88","Type":"ContainerDied","Data":"5cb9d770b3ba62381673fc0417d3e562b4b79209aa25ef6b3e024272415006ae"} Oct 30 18:40:17.009119 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:17.008980 4603 scope.go:122] "RemoveContainer" containerID="dd9711d5b66007469ce86d91263f92ac85c307a18897fff91c3361931f121022" Oct 30 18:40:17.028465 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:17.028433 4603 scope.go:122] "RemoveContainer" containerID="dd9711d5b66007469ce86d91263f92ac85c307a18897fff91c3361931f121022" Oct 30 18:40:17.028821 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:40:17.028802 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd9711d5b66007469ce86d91263f92ac85c307a18897fff91c3361931f121022\": container with ID starting with dd9711d5b66007469ce86d91263f92ac85c307a18897fff91c3361931f121022 not found: ID does not exist" containerID="dd9711d5b66007469ce86d91263f92ac85c307a18897fff91c3361931f121022" Oct 30 18:40:17.028890 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:17.028843 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd9711d5b66007469ce86d91263f92ac85c307a18897fff91c3361931f121022"} err="failed to get container status \"dd9711d5b66007469ce86d91263f92ac85c307a18897fff91c3361931f121022\": rpc error: code = NotFound desc = could not find container \"dd9711d5b66007469ce86d91263f92ac85c307a18897fff91c3361931f121022\": container with ID starting with dd9711d5b66007469ce86d91263f92ac85c307a18897fff91c3361931f121022 not found: ID does not exist" Oct 30 18:40:17.067646 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:17.067549 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt"] Oct 30 18:40:17.077003 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:17.076976 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-qj7mt"] Oct 30 18:40:17.080358 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:17.080328 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:17.409979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:17.409946 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9bcc5bf-0143-429f-95c1-298df70d5f88" path="/var/lib/kubelet/pods/f9bcc5bf-0143-429f-95c1-298df70d5f88/volumes" Oct 30 18:40:17.814865 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:17.814836 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:18.005438 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:18.005409 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:18.175323 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:18.175255 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8"] Oct 30 18:40:18.175516 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:18.175480 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" podUID="73366d41-4910-43f0-aad8-2542401ec0f6" containerName="istio-proxy" containerID="cri-o://0088043c96740ee34b18139e21b425d3d572db30d303cfb0003cc5fb9539fba7" gracePeriod=30 Oct 30 18:40:18.226135 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:18.226103 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:18.592529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:18.591561 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:18.753927 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:18.753893 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:19.404633 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:19.404599 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:20.592527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:20.583242 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:21.930133 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:40:21.930099 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:40:22.365242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:22.365200 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:22.788435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:22.788400 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:22.955666 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:22.955630 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:23.597446 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.597415 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:40:23.673422 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.673394 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq"] Oct 30 18:40:23.706301 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.706273 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73366d41-4910-43f0-aad8-2542401ec0f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73366d41-4910-43f0-aad8-2542401ec0f6-istio-podinfo\") pod \"73366d41-4910-43f0-aad8-2542401ec0f6\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " Oct 30 18:40:23.706464 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.706318 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-credential-socket\") pod \"73366d41-4910-43f0-aad8-2542401ec0f6\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " Oct 30 18:40:23.706464 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.706365 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73366d41-4910-43f0-aad8-2542401ec0f6-kube-api-access-dwpbr\" (UniqueName: \"kubernetes.io/projected/73366d41-4910-43f0-aad8-2542401ec0f6-kube-api-access-dwpbr\") pod \"73366d41-4910-43f0-aad8-2542401ec0f6\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " Oct 30 18:40:23.706464 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.706405 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-workload-certs\") pod \"73366d41-4910-43f0-aad8-2542401ec0f6\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " Oct 30 18:40:23.706654 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.706478 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-istio-envoy\") pod \"73366d41-4910-43f0-aad8-2542401ec0f6\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " Oct 30 18:40:23.706654 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.706586 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73366d41-4910-43f0-aad8-2542401ec0f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73366d41-4910-43f0-aad8-2542401ec0f6-istiod-ca-cert\") pod \"73366d41-4910-43f0-aad8-2542401ec0f6\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " Oct 30 18:40:23.706784 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.706678 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-workload-socket\") pod \"73366d41-4910-43f0-aad8-2542401ec0f6\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " Oct 30 18:40:23.706784 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.706706 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-istio-data\") pod \"73366d41-4910-43f0-aad8-2542401ec0f6\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " Oct 30 18:40:23.706784 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.706735 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-credential-socket" pod "73366d41-4910-43f0-aad8-2542401ec0f6" (UID: "73366d41-4910-43f0-aad8-2542401ec0f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:23.706784 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.706769 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73366d41-4910-43f0-aad8-2542401ec0f6-istio-token\" (UniqueName: \"kubernetes.io/projected/73366d41-4910-43f0-aad8-2542401ec0f6-istio-token\") pod \"73366d41-4910-43f0-aad8-2542401ec0f6\" (UID: \"73366d41-4910-43f0-aad8-2542401ec0f6\") " Oct 30 18:40:23.707003 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.706788 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-workload-certs" pod "73366d41-4910-43f0-aad8-2542401ec0f6" (UID: "73366d41-4910-43f0-aad8-2542401ec0f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:23.707003 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.706907 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-workload-socket" pod "73366d41-4910-43f0-aad8-2542401ec0f6" (UID: "73366d41-4910-43f0-aad8-2542401ec0f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:23.707003 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.706989 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-istio-data" pod "73366d41-4910-43f0-aad8-2542401ec0f6" (UID: "73366d41-4910-43f0-aad8-2542401ec0f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:23.707161 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.707019 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73366d41-4910-43f0-aad8-2542401ec0f6-istiod-ca-cert" pod "73366d41-4910-43f0-aad8-2542401ec0f6" (UID: "73366d41-4910-43f0-aad8-2542401ec0f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:23.707707 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.707683 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:23.707840 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.707710 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:23.707840 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.707726 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:23.707840 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.707741 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:23.707840 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.707754 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73366d41-4910-43f0-aad8-2542401ec0f6-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:23.709000 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.708970 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73366d41-4910-43f0-aad8-2542401ec0f6-kube-api-access-dwpbr" pod "73366d41-4910-43f0-aad8-2542401ec0f6" (UID: "73366d41-4910-43f0-aad8-2542401ec0f6"). InnerVolumeSpecName "kube-api-access-dwpbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:23.709179 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.709160 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73366d41-4910-43f0-aad8-2542401ec0f6-istio-token" pod "73366d41-4910-43f0-aad8-2542401ec0f6" (UID: "73366d41-4910-43f0-aad8-2542401ec0f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:23.709458 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.709426 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73366d41-4910-43f0-aad8-2542401ec0f6-istio-podinfo" pod "73366d41-4910-43f0-aad8-2542401ec0f6" (UID: "73366d41-4910-43f0-aad8-2542401ec0f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:40:23.709595 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.709514 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-istio-envoy" pod "73366d41-4910-43f0-aad8-2542401ec0f6" (UID: "73366d41-4910-43f0-aad8-2542401ec0f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:23.809439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.809410 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73366d41-4910-43f0-aad8-2542401ec0f6-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:23.809439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.809436 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73366d41-4910-43f0-aad8-2542401ec0f6-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:23.809439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.809445 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73366d41-4910-43f0-aad8-2542401ec0f6-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:23.809749 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.809454 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwpbr\" (UniqueName: \"kubernetes.io/projected/73366d41-4910-43f0-aad8-2542401ec0f6-kube-api-access-dwpbr\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:23.822522 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:23.822474 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:24.053439 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:24.053408 4603 generic.go:356] "Generic (PLEG): container finished" podID="73366d41-4910-43f0-aad8-2542401ec0f6" containerID="0088043c96740ee34b18139e21b425d3d572db30d303cfb0003cc5fb9539fba7" exitCode=0 Oct 30 18:40:24.053979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:24.053455 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" event={"ID":"73366d41-4910-43f0-aad8-2542401ec0f6","Type":"ContainerDied","Data":"0088043c96740ee34b18139e21b425d3d572db30d303cfb0003cc5fb9539fba7"} Oct 30 18:40:24.053979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:24.053480 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" Oct 30 18:40:24.053979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:24.053494 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8" event={"ID":"73366d41-4910-43f0-aad8-2542401ec0f6","Type":"ContainerDied","Data":"19ddfd1635b62dd96160a04887e1a6b982cc9f8bd892dce48c91772adac8bd1f"} Oct 30 18:40:24.053979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:24.053534 4603 scope.go:122] "RemoveContainer" containerID="0088043c96740ee34b18139e21b425d3d572db30d303cfb0003cc5fb9539fba7" Oct 30 18:40:24.067211 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:24.067191 4603 scope.go:122] "RemoveContainer" containerID="d95c310ab51db3b9a868be677d78cc335b2b18e6ac980e855881ec4e704755d7" Oct 30 18:40:24.076239 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:24.076216 4603 scope.go:122] "RemoveContainer" containerID="0088043c96740ee34b18139e21b425d3d572db30d303cfb0003cc5fb9539fba7" Oct 30 18:40:24.076538 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:40:24.076490 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0088043c96740ee34b18139e21b425d3d572db30d303cfb0003cc5fb9539fba7\": container with ID starting with 0088043c96740ee34b18139e21b425d3d572db30d303cfb0003cc5fb9539fba7 not found: ID does not exist" containerID="0088043c96740ee34b18139e21b425d3d572db30d303cfb0003cc5fb9539fba7" Oct 30 18:40:24.076662 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:24.076547 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0088043c96740ee34b18139e21b425d3d572db30d303cfb0003cc5fb9539fba7"} err="failed to get container status \"0088043c96740ee34b18139e21b425d3d572db30d303cfb0003cc5fb9539fba7\": rpc error: code = NotFound desc = could not find container \"0088043c96740ee34b18139e21b425d3d572db30d303cfb0003cc5fb9539fba7\": container with ID starting with 0088043c96740ee34b18139e21b425d3d572db30d303cfb0003cc5fb9539fba7 not found: ID does not exist" Oct 30 18:40:24.076662 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:24.076565 4603 scope.go:122] "RemoveContainer" containerID="d95c310ab51db3b9a868be677d78cc335b2b18e6ac980e855881ec4e704755d7" Oct 30 18:40:24.076877 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:40:24.076859 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d95c310ab51db3b9a868be677d78cc335b2b18e6ac980e855881ec4e704755d7\": container with ID starting with d95c310ab51db3b9a868be677d78cc335b2b18e6ac980e855881ec4e704755d7 not found: ID does not exist" containerID="d95c310ab51db3b9a868be677d78cc335b2b18e6ac980e855881ec4e704755d7" Oct 30 18:40:24.076939 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:24.076887 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d95c310ab51db3b9a868be677d78cc335b2b18e6ac980e855881ec4e704755d7"} err="failed to get container status \"d95c310ab51db3b9a868be677d78cc335b2b18e6ac980e855881ec4e704755d7\": rpc error: code = NotFound desc = could not find container \"d95c310ab51db3b9a868be677d78cc335b2b18e6ac980e855881ec4e704755d7\": container with ID starting with d95c310ab51db3b9a868be677d78cc335b2b18e6ac980e855881ec4e704755d7 not found: ID does not exist" Oct 30 18:40:24.105663 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:24.105633 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8"] Oct 30 18:40:24.113633 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:24.113601 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-gcqm8"] Oct 30 18:40:24.145924 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:24.145897 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:25.409858 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:25.409829 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73366d41-4910-43f0-aad8-2542401ec0f6" path="/var/lib/kubelet/pods/73366d41-4910-43f0-aad8-2542401ec0f6/volumes" Oct 30 18:40:25.753252 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:25.753165 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:27.439774 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:27.439726 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:29.270448 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:29.270396 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:29.722552 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:29.722438 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:29.870627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:29.870595 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:31.059772 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:31.059736 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:31.888686 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:31.888643 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" containerID="cri-o://74b6b90807e273cd6f183f0ee624b7d5461d694fd7caaa37c9109a3d645dd181" gracePeriod=45 Oct 30 18:40:31.908076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:31.908040 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" containerID="cri-o://f31f9568e89352f44f7d7b8c70b7eaca385631b7c58ced41b612b1a3a1e25730" gracePeriod=45 Oct 30 18:40:32.120477 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.120444 4603 generic.go:356] "Generic (PLEG): container finished" podID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerID="f31f9568e89352f44f7d7b8c70b7eaca385631b7c58ced41b612b1a3a1e25730" exitCode=0 Oct 30 18:40:32.120923 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.120640 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" event={"ID":"2d8dda47-f665-4fcb-bdae-3430e419841c","Type":"ContainerDied","Data":"f31f9568e89352f44f7d7b8c70b7eaca385631b7c58ced41b612b1a3a1e25730"} Oct 30 18:40:32.122298 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.122272 4603 generic.go:356] "Generic (PLEG): container finished" podID="521cc860-6923-45a7-8823-4859624c9267" containerID="74b6b90807e273cd6f183f0ee624b7d5461d694fd7caaa37c9109a3d645dd181" exitCode=0 Oct 30 18:40:32.122422 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.122340 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" event={"ID":"521cc860-6923-45a7-8823-4859624c9267","Type":"ContainerDied","Data":"74b6b90807e273cd6f183f0ee624b7d5461d694fd7caaa37c9109a3d645dd181"} Oct 30 18:40:32.182555 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.180819 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:40:32.203981 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.203956 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:40:32.328986 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.328956 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/521cc860-6923-45a7-8823-4859624c9267-kube-api-access-gb89j\" (UniqueName: \"kubernetes.io/projected/521cc860-6923-45a7-8823-4859624c9267-kube-api-access-gb89j\") pod \"521cc860-6923-45a7-8823-4859624c9267\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " Oct 30 18:40:32.329188 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.329006 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-home\" (UniqueName: \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-home\") pod \"521cc860-6923-45a7-8823-4859624c9267\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " Oct 30 18:40:32.329188 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.329050 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-tmp-dir\") pod \"2d8dda47-f665-4fcb-bdae-3430e419841c\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " Oct 30 18:40:32.329188 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.329172 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-model-cache\") pod \"521cc860-6923-45a7-8823-4859624c9267\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " Oct 30 18:40:32.329410 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.329199 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-home\" (UniqueName: \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-home\") pod \"2d8dda47-f665-4fcb-bdae-3430e419841c\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " Oct 30 18:40:32.329410 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.329385 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/521cc860-6923-45a7-8823-4859624c9267-tls-certs\" (UniqueName: \"kubernetes.io/secret/521cc860-6923-45a7-8823-4859624c9267-tls-certs\") pod \"521cc860-6923-45a7-8823-4859624c9267\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " Oct 30 18:40:32.329577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.329460 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-model-cache\") pod \"2d8dda47-f665-4fcb-bdae-3430e419841c\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " Oct 30 18:40:32.329577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.329460 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-tmp-dir" pod "2d8dda47-f665-4fcb-bdae-3430e419841c" (UID: "2d8dda47-f665-4fcb-bdae-3430e419841c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:32.329696 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.329600 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-dshm\" (UniqueName: \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-dshm\") pod \"521cc860-6923-45a7-8823-4859624c9267\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " Oct 30 18:40:32.329696 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.329645 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8dda47-f665-4fcb-bdae-3430e419841c-kube-api-access-88mgj\" (UniqueName: \"kubernetes.io/projected/2d8dda47-f665-4fcb-bdae-3430e419841c-kube-api-access-88mgj\") pod \"2d8dda47-f665-4fcb-bdae-3430e419841c\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " Oct 30 18:40:32.329816 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.329716 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-model-cache" pod "2d8dda47-f665-4fcb-bdae-3430e419841c" (UID: "2d8dda47-f665-4fcb-bdae-3430e419841c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:32.329816 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.329730 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d8dda47-f665-4fcb-bdae-3430e419841c-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d8dda47-f665-4fcb-bdae-3430e419841c-tls-certs\") pod \"2d8dda47-f665-4fcb-bdae-3430e419841c\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " Oct 30 18:40:32.329816 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.329759 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-tmp-dir\") pod \"521cc860-6923-45a7-8823-4859624c9267\" (UID: \"521cc860-6923-45a7-8823-4859624c9267\") " Oct 30 18:40:32.329816 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.329796 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-dshm\") pod \"2d8dda47-f665-4fcb-bdae-3430e419841c\" (UID: \"2d8dda47-f665-4fcb-bdae-3430e419841c\") " Oct 30 18:40:32.330294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.330225 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-home" pod "2d8dda47-f665-4fcb-bdae-3430e419841c" (UID: "2d8dda47-f665-4fcb-bdae-3430e419841c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:32.332174 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.330558 4603 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-tmp-dir\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.332174 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.330583 4603 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-home\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.332174 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.330599 4603 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-model-cache\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.332174 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.331066 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-model-cache" pod "521cc860-6923-45a7-8823-4859624c9267" (UID: "521cc860-6923-45a7-8823-4859624c9267"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:32.332174 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.332113 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-tmp-dir" pod "521cc860-6923-45a7-8823-4859624c9267" (UID: "521cc860-6923-45a7-8823-4859624c9267"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:32.332703 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.332668 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-home" pod "521cc860-6923-45a7-8823-4859624c9267" (UID: "521cc860-6923-45a7-8823-4859624c9267"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:32.335084 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.335047 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8dda47-f665-4fcb-bdae-3430e419841c-kube-api-access-88mgj" pod "2d8dda47-f665-4fcb-bdae-3430e419841c" (UID: "2d8dda47-f665-4fcb-bdae-3430e419841c"). InnerVolumeSpecName "kube-api-access-88mgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:32.335384 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.335211 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-dshm" pod "2d8dda47-f665-4fcb-bdae-3430e419841c" (UID: "2d8dda47-f665-4fcb-bdae-3430e419841c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:32.335384 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.335298 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-dshm" pod "521cc860-6923-45a7-8823-4859624c9267" (UID: "521cc860-6923-45a7-8823-4859624c9267"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:32.335384 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.335341 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/521cc860-6923-45a7-8823-4859624c9267-tls-certs" pod "521cc860-6923-45a7-8823-4859624c9267" (UID: "521cc860-6923-45a7-8823-4859624c9267"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:32.335643 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.335619 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521cc860-6923-45a7-8823-4859624c9267-kube-api-access-gb89j" pod "521cc860-6923-45a7-8823-4859624c9267" (UID: "521cc860-6923-45a7-8823-4859624c9267"). InnerVolumeSpecName "kube-api-access-gb89j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:32.335938 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.335905 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d8dda47-f665-4fcb-bdae-3430e419841c-tls-certs" pod "2d8dda47-f665-4fcb-bdae-3430e419841c" (UID: "2d8dda47-f665-4fcb-bdae-3430e419841c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:32.431556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.431463 4603 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d8dda47-f665-4fcb-bdae-3430e419841c-tls-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.431556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.431489 4603 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-tmp-dir\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.431556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.431497 4603 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d8dda47-f665-4fcb-bdae-3430e419841c-dshm\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.431556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.431532 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb89j\" (UniqueName: \"kubernetes.io/projected/521cc860-6923-45a7-8823-4859624c9267-kube-api-access-gb89j\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.431556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.431542 4603 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-home\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.431556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.431550 4603 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-model-cache\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.431556 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.431559 4603 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/521cc860-6923-45a7-8823-4859624c9267-tls-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.431893 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.431566 4603 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/521cc860-6923-45a7-8823-4859624c9267-dshm\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.431893 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.431574 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88mgj\" (UniqueName: \"kubernetes.io/projected/2d8dda47-f665-4fcb-bdae-3430e419841c-kube-api-access-88mgj\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.812062 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:32.811950 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:33.129453 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:33.129377 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" Oct 30 18:40:33.129453 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:33.129406 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2" event={"ID":"521cc860-6923-45a7-8823-4859624c9267","Type":"ContainerDied","Data":"5d90a2d95edd73ca43b50d846fa6c22e01ac043f6ce0e7f52a5f42930a8ad50f"} Oct 30 18:40:33.129453 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:33.129459 4603 scope.go:122] "RemoveContainer" containerID="74b6b90807e273cd6f183f0ee624b7d5461d694fd7caaa37c9109a3d645dd181" Oct 30 18:40:33.131062 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:33.131041 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" event={"ID":"2d8dda47-f665-4fcb-bdae-3430e419841c","Type":"ContainerDied","Data":"8696aa34c47ffce26b25ae19f21a6a9aa9c0c616735feaa6a5bb0da108fd32e6"} Oct 30 18:40:33.131183 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:33.131061 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z" Oct 30 18:40:33.139897 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:33.139875 4603 scope.go:122] "RemoveContainer" containerID="f31f9568e89352f44f7d7b8c70b7eaca385631b7c58ced41b612b1a3a1e25730" Oct 30 18:40:33.179850 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:33.179825 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z"] Oct 30 18:40:33.191584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:33.191560 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/premium-w3-3cc458-kserve-6554499db7-7zf2z"] Oct 30 18:40:33.217929 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:33.217907 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2"] Oct 30 18:40:33.221697 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:33.221679 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/distinct-w3-3cc458-kserve-c9d86cb78-ps8j2"] Oct 30 18:40:33.409860 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:33.409793 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" path="/var/lib/kubelet/pods/2d8dda47-f665-4fcb-bdae-3430e419841c/volumes" Oct 30 18:40:33.410227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:33.410212 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="521cc860-6923-45a7-8823-4859624c9267" path="/var/lib/kubelet/pods/521cc860-6923-45a7-8823-4859624c9267/volumes" Oct 30 18:40:34.761542 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:34.761469 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:36.460004 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:36.459967 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:38.150818 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:38.150783 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:38.695386 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:38.695352 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" containerID="cri-o://a37f1ac4eaa4cef7dea486ac59ac985bacbd9661f05e750063ce8036c5b151d9" gracePeriod=45 Oct 30 18:40:38.951845 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:38.951791 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:40:39.011887 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.011854 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-dshm\") pod \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " Oct 30 18:40:39.012073 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.011945 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-tmp-dir\") pod \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " Oct 30 18:40:39.012192 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.012082 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-model-cache\") pod \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " Oct 30 18:40:39.012192 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.012147 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-home\" (UniqueName: \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-home\") pod \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " Oct 30 18:40:39.012309 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.012202 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-tls-certs\") pod \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " Oct 30 18:40:39.012309 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.012242 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-tmp-dir" pod "1e0d5196-98c7-4b7c-b045-623b42c2e3d0" (UID: "1e0d5196-98c7-4b7c-b045-623b42c2e3d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:39.012309 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.012284 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-kube-api-access-zcss4\" (UniqueName: \"kubernetes.io/projected/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-kube-api-access-zcss4\") pod \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\" (UID: \"1e0d5196-98c7-4b7c-b045-623b42c2e3d0\") " Oct 30 18:40:39.012491 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.012473 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-home" pod "1e0d5196-98c7-4b7c-b045-623b42c2e3d0" (UID: "1e0d5196-98c7-4b7c-b045-623b42c2e3d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:39.012747 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.012721 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-model-cache" pod "1e0d5196-98c7-4b7c-b045-623b42c2e3d0" (UID: "1e0d5196-98c7-4b7c-b045-623b42c2e3d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:39.012856 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.012833 4603 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-tmp-dir\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:39.012924 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.012863 4603 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-model-cache\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:39.012924 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.012875 4603 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-home\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:39.014780 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.014749 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-dshm" pod "1e0d5196-98c7-4b7c-b045-623b42c2e3d0" (UID: "1e0d5196-98c7-4b7c-b045-623b42c2e3d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:39.015335 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.015308 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-kube-api-access-zcss4" pod "1e0d5196-98c7-4b7c-b045-623b42c2e3d0" (UID: "1e0d5196-98c7-4b7c-b045-623b42c2e3d0"). InnerVolumeSpecName "kube-api-access-zcss4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:39.015427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.015319 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-tls-certs" pod "1e0d5196-98c7-4b7c-b045-623b42c2e3d0" (UID: "1e0d5196-98c7-4b7c-b045-623b42c2e3d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:39.114294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.114252 4603 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-tls-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:39.114294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.114294 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcss4\" (UniqueName: \"kubernetes.io/projected/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-kube-api-access-zcss4\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:39.114604 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.114311 4603 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e0d5196-98c7-4b7c-b045-623b42c2e3d0-dshm\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:39.187387 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.187357 4603 generic.go:356] "Generic (PLEG): container finished" podID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerID="a37f1ac4eaa4cef7dea486ac59ac985bacbd9661f05e750063ce8036c5b151d9" exitCode=0 Oct 30 18:40:39.187857 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.187413 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" Oct 30 18:40:39.187857 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.187440 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" event={"ID":"1e0d5196-98c7-4b7c-b045-623b42c2e3d0","Type":"ContainerDied","Data":"a37f1ac4eaa4cef7dea486ac59ac985bacbd9661f05e750063ce8036c5b151d9"} Oct 30 18:40:39.187857 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.187490 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq" event={"ID":"1e0d5196-98c7-4b7c-b045-623b42c2e3d0","Type":"ContainerDied","Data":"6043ca5f32e0efbc365ed7aec4e6e58d9630a261ee14d06afa58a3279caf4870"} Oct 30 18:40:39.187857 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.187533 4603 scope.go:122] "RemoveContainer" containerID="a37f1ac4eaa4cef7dea486ac59ac985bacbd9661f05e750063ce8036c5b151d9" Oct 30 18:40:39.198338 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.198321 4603 scope.go:122] "RemoveContainer" containerID="a37f1ac4eaa4cef7dea486ac59ac985bacbd9661f05e750063ce8036c5b151d9" Oct 30 18:40:39.198643 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:40:39.198620 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a37f1ac4eaa4cef7dea486ac59ac985bacbd9661f05e750063ce8036c5b151d9\": container with ID starting with a37f1ac4eaa4cef7dea486ac59ac985bacbd9661f05e750063ce8036c5b151d9 not found: ID does not exist" containerID="a37f1ac4eaa4cef7dea486ac59ac985bacbd9661f05e750063ce8036c5b151d9" Oct 30 18:40:39.198698 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.198652 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a37f1ac4eaa4cef7dea486ac59ac985bacbd9661f05e750063ce8036c5b151d9"} err="failed to get container status \"a37f1ac4eaa4cef7dea486ac59ac985bacbd9661f05e750063ce8036c5b151d9\": rpc error: code = NotFound desc = could not find container \"a37f1ac4eaa4cef7dea486ac59ac985bacbd9661f05e750063ce8036c5b151d9\": container with ID starting with a37f1ac4eaa4cef7dea486ac59ac985bacbd9661f05e750063ce8036c5b151d9 not found: ID does not exist" Oct 30 18:40:39.238340 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.238275 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq"] Oct 30 18:40:39.251079 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.251056 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-419d32/premium-w4-419d32-kserve-6b55bf5ccf-q7ddq"] Oct 30 18:40:39.410294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.410266 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" path="/var/lib/kubelet/pods/1e0d5196-98c7-4b7c-b045-623b42c2e3d0/volumes" Oct 30 18:40:39.855587 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:39.854935 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:41.111800 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:41.111769 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn"] Oct 30 18:40:41.112336 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:41.112027 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" podUID="e5c8b1f8-fba8-4afc-8887-8b1375d67b22" containerName="maas-api" containerID="cri-o://764446c320c5550776b951f1283da60ce87998c86f8b69f0f4859a38b676db69" gracePeriod=30 Oct 30 18:40:41.212513 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:41.212480 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:41.513527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:41.500531 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" Oct 30 18:40:41.570563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:41.567136 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-openshift-service-ca\") pod \"e5c8b1f8-fba8-4afc-8887-8b1375d67b22\" (UID: \"e5c8b1f8-fba8-4afc-8887-8b1375d67b22\") " Oct 30 18:40:41.570563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:41.567408 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-kube-api-access-j86mv\" (UniqueName: \"kubernetes.io/projected/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-kube-api-access-j86mv\") pod \"e5c8b1f8-fba8-4afc-8887-8b1375d67b22\" (UID: \"e5c8b1f8-fba8-4afc-8887-8b1375d67b22\") " Oct 30 18:40:41.570563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:41.567449 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-metrics-certs\") pod \"e5c8b1f8-fba8-4afc-8887-8b1375d67b22\" (UID: \"e5c8b1f8-fba8-4afc-8887-8b1375d67b22\") " Oct 30 18:40:41.570563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:41.567484 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-maas-api-tls\") pod \"e5c8b1f8-fba8-4afc-8887-8b1375d67b22\" (UID: \"e5c8b1f8-fba8-4afc-8887-8b1375d67b22\") " Oct 30 18:40:41.572396 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:41.572348 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-openshift-service-ca" pod "e5c8b1f8-fba8-4afc-8887-8b1375d67b22" (UID: "e5c8b1f8-fba8-4afc-8887-8b1375d67b22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:41.574043 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:41.574008 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-kube-api-access-j86mv" pod "e5c8b1f8-fba8-4afc-8887-8b1375d67b22" (UID: "e5c8b1f8-fba8-4afc-8887-8b1375d67b22"). InnerVolumeSpecName "kube-api-access-j86mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:41.576701 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:41.576663 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-maas-api-tls" pod "e5c8b1f8-fba8-4afc-8887-8b1375d67b22" (UID: "e5c8b1f8-fba8-4afc-8887-8b1375d67b22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:41.609207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:41.595397 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-metrics-certs" pod "e5c8b1f8-fba8-4afc-8887-8b1375d67b22" (UID: "e5c8b1f8-fba8-4afc-8887-8b1375d67b22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:41.675762 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:41.675677 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j86mv\" (UniqueName: \"kubernetes.io/projected/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-kube-api-access-j86mv\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:41.675762 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:41.675711 4603 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:41.675762 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:41.675724 4603 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-maas-api-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:41.675762 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:41.675736 4603 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c8b1f8-fba8-4afc-8887-8b1375d67b22-openshift-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:41.815233 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:41.815195 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:41.969844 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:41.969754 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:42.117545 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:42.117486 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:42.207338 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:42.207310 4603 generic.go:356] "Generic (PLEG): container finished" podID="e5c8b1f8-fba8-4afc-8887-8b1375d67b22" containerID="764446c320c5550776b951f1283da60ce87998c86f8b69f0f4859a38b676db69" exitCode=0 Oct 30 18:40:42.207484 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:42.207358 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" Oct 30 18:40:42.207484 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:42.207389 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" event={"ID":"e5c8b1f8-fba8-4afc-8887-8b1375d67b22","Type":"ContainerDied","Data":"764446c320c5550776b951f1283da60ce87998c86f8b69f0f4859a38b676db69"} Oct 30 18:40:42.207484 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:42.207442 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn" event={"ID":"e5c8b1f8-fba8-4afc-8887-8b1375d67b22","Type":"ContainerDied","Data":"9543187e0b903fe1ffe1f366b05342c0f9e53eb43fb78a9f52c509d7c24e4f3c"} Oct 30 18:40:42.207484 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:42.207465 4603 scope.go:122] "RemoveContainer" containerID="764446c320c5550776b951f1283da60ce87998c86f8b69f0f4859a38b676db69" Oct 30 18:40:42.216922 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:42.216907 4603 scope.go:122] "RemoveContainer" containerID="764446c320c5550776b951f1283da60ce87998c86f8b69f0f4859a38b676db69" Oct 30 18:40:42.217181 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:40:42.217165 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"764446c320c5550776b951f1283da60ce87998c86f8b69f0f4859a38b676db69\": container with ID starting with 764446c320c5550776b951f1283da60ce87998c86f8b69f0f4859a38b676db69 not found: ID does not exist" containerID="764446c320c5550776b951f1283da60ce87998c86f8b69f0f4859a38b676db69" Oct 30 18:40:42.217227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:42.217189 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"764446c320c5550776b951f1283da60ce87998c86f8b69f0f4859a38b676db69"} err="failed to get container status \"764446c320c5550776b951f1283da60ce87998c86f8b69f0f4859a38b676db69\": rpc error: code = NotFound desc = could not find container \"764446c320c5550776b951f1283da60ce87998c86f8b69f0f4859a38b676db69\": container with ID starting with 764446c320c5550776b951f1283da60ce87998c86f8b69f0f4859a38b676db69 not found: ID does not exist" Oct 30 18:40:42.257251 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:42.257196 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn"] Oct 30 18:40:42.261812 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:42.261793 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3cc458-949f89fc-f58qn"] Oct 30 18:40:42.508590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:42.508476 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:43.409416 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:43.409384 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5c8b1f8-fba8-4afc-8887-8b1375d67b22" path="/var/lib/kubelet/pods/e5c8b1f8-fba8-4afc-8887-8b1375d67b22/volumes" Oct 30 18:40:43.953704 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:43.953664 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:45.586560 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:45.586449 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:46.949458 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:46.949425 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:47.636884 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:47.636846 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:47.779726 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:47.779691 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:48.213295 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:48.213257 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:49.640664 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:49.640633 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:49.910767 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:49.910643 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:50.068745 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:50.068711 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:50.223287 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:50.223197 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn"] Oct 30 18:40:50.223463 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:50.223438 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" podUID="22e0b280-2b65-4fad-8344-67049190d108" containerName="istio-proxy" containerID="cri-o://987b8014d0d4886b3ebdf1c880a7bb694d1169d0fd45406d4d558b7cc98eda45" gracePeriod=30 Oct 30 18:40:50.227660 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:50.227638 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x"] Oct 30 18:40:50.227843 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:50.227825 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" podUID="1d187b51-f5c6-4a63-8300-2f01184a15d3" containerName="istio-proxy" containerID="cri-o://f65aac4dd93bea107473a1364b0f53516964b3092ac1cdbb6cfbf7db84119f14" gracePeriod=30 Oct 30 18:40:50.232992 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:50.232966 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc"] Oct 30 18:40:50.233168 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:50.233149 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" podUID="b664f4ca-736e-4cb1-991d-985d1e6aff34" containerName="istio-proxy" containerID="cri-o://80d5343a4d7fab6f7441d8100515aaec7fabac590594da9544683a876a70c15d" gracePeriod=30 Oct 30 18:40:50.237154 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:50.237132 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v"] Oct 30 18:40:50.237322 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:50.237304 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" podUID="d4a3adf6-fb23-4334-8bab-f800b4e8e089" containerName="istio-proxy" containerID="cri-o://3fb29526621b83ed763d3401b3b37a008321bb644366f41e7cbc405cadeef19c" gracePeriod=30 Oct 30 18:40:50.284232 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:50.284199 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:51.452061 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:51.452030 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:51.928956 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:40:51.928919 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:40:53.077644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:53.077605 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:53.278442 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:53.278411 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:54.248585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:54.248547 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:40:54.852266 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:54.852236 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:55.677237 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.676919 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:40:55.775528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.771918 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-istio-envoy\") pod \"22e0b280-2b65-4fad-8344-67049190d108\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " Oct 30 18:40:55.775528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.771975 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-istio-data\") pod \"22e0b280-2b65-4fad-8344-67049190d108\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " Oct 30 18:40:55.775528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.772092 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-workload-certs\") pod \"22e0b280-2b65-4fad-8344-67049190d108\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " Oct 30 18:40:55.775528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.772131 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22e0b280-2b65-4fad-8344-67049190d108-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e0b280-2b65-4fad-8344-67049190d108-istio-podinfo\") pod \"22e0b280-2b65-4fad-8344-67049190d108\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " Oct 30 18:40:55.775528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.772172 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22e0b280-2b65-4fad-8344-67049190d108-istio-token\" (UniqueName: \"kubernetes.io/projected/22e0b280-2b65-4fad-8344-67049190d108-istio-token\") pod \"22e0b280-2b65-4fad-8344-67049190d108\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " Oct 30 18:40:55.775528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.772232 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-workload-socket\") pod \"22e0b280-2b65-4fad-8344-67049190d108\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " Oct 30 18:40:55.775528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.772381 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22e0b280-2b65-4fad-8344-67049190d108-kube-api-access-rk8wl\" (UniqueName: \"kubernetes.io/projected/22e0b280-2b65-4fad-8344-67049190d108-kube-api-access-rk8wl\") pod \"22e0b280-2b65-4fad-8344-67049190d108\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " Oct 30 18:40:55.775528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.772425 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22e0b280-2b65-4fad-8344-67049190d108-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e0b280-2b65-4fad-8344-67049190d108-istiod-ca-cert\") pod \"22e0b280-2b65-4fad-8344-67049190d108\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " Oct 30 18:40:55.775528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.772473 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-credential-socket\") pod \"22e0b280-2b65-4fad-8344-67049190d108\" (UID: \"22e0b280-2b65-4fad-8344-67049190d108\") " Oct 30 18:40:55.900537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.796589 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-credential-socket" pod "22e0b280-2b65-4fad-8344-67049190d108" (UID: "22e0b280-2b65-4fad-8344-67049190d108"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:55.900537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.816589 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-workload-socket" pod "22e0b280-2b65-4fad-8344-67049190d108" (UID: "22e0b280-2b65-4fad-8344-67049190d108"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:55.900537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.816699 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-workload-certs" pod "22e0b280-2b65-4fad-8344-67049190d108" (UID: "22e0b280-2b65-4fad-8344-67049190d108"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:55.900537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.829733 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e0b280-2b65-4fad-8344-67049190d108-istiod-ca-cert" pod "22e0b280-2b65-4fad-8344-67049190d108" (UID: "22e0b280-2b65-4fad-8344-67049190d108"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:55.900537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.889929 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.900537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.889968 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.900537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.889985 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e0b280-2b65-4fad-8344-67049190d108-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.900537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.889999 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.900537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.899646 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-istio-data" pod "22e0b280-2b65-4fad-8344-67049190d108" (UID: "22e0b280-2b65-4fad-8344-67049190d108"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:55.969718 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.966683 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e0b280-2b65-4fad-8344-67049190d108-istio-token" pod "22e0b280-2b65-4fad-8344-67049190d108" (UID: "22e0b280-2b65-4fad-8344-67049190d108"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:55.970134 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.970089 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e0b280-2b65-4fad-8344-67049190d108-kube-api-access-rk8wl" pod "22e0b280-2b65-4fad-8344-67049190d108" (UID: "22e0b280-2b65-4fad-8344-67049190d108"). InnerVolumeSpecName "kube-api-access-rk8wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:55.996313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.996234 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk8wl\" (UniqueName: \"kubernetes.io/projected/22e0b280-2b65-4fad-8344-67049190d108-kube-api-access-rk8wl\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.996313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.996270 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.996313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:55.996287 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e0b280-2b65-4fad-8344-67049190d108-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.010020 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.009847 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-istio-envoy" pod "22e0b280-2b65-4fad-8344-67049190d108" (UID: "22e0b280-2b65-4fad-8344-67049190d108"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:56.010020 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.009976 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22e0b280-2b65-4fad-8344-67049190d108-istio-podinfo" pod "22e0b280-2b65-4fad-8344-67049190d108" (UID: "22e0b280-2b65-4fad-8344-67049190d108"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:40:56.050535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.047087 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:56.110745 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.110702 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e0b280-2b65-4fad-8344-67049190d108-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.146242 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.146207 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e0b280-2b65-4fad-8344-67049190d108-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.306661 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.288735 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:56.415698 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.414775 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw"] Oct 30 18:40:56.415698 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.415024 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" podUID="47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" containerName="istio-proxy" containerID="cri-o://88a9eaba348c52a304694c657c04c62b48cd9d534a7106e34e43a3fe117a1905" gracePeriod=30 Oct 30 18:40:56.441332 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.441240 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:56.491918 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.483607 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc"] Oct 30 18:40:56.491918 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.483911 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" podUID="d5567046-aa24-421a-8339-fdd9cf56e62c" containerName="istio-proxy" containerID="cri-o://ac2d2ba01f24fe73bc3985faf348fec2383e49bb1f192b3800f6546e07025dfc" gracePeriod=30 Oct 30 18:40:56.521996 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.518243 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:40:56.558361 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.545872 4603 generic.go:356] "Generic (PLEG): container finished" podID="1d187b51-f5c6-4a63-8300-2f01184a15d3" containerID="f65aac4dd93bea107473a1364b0f53516964b3092ac1cdbb6cfbf7db84119f14" exitCode=0 Oct 30 18:40:56.558361 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.546078 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" event={"ID":"1d187b51-f5c6-4a63-8300-2f01184a15d3","Type":"ContainerDied","Data":"f65aac4dd93bea107473a1364b0f53516964b3092ac1cdbb6cfbf7db84119f14"} Oct 30 18:40:56.558361 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.546134 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" event={"ID":"1d187b51-f5c6-4a63-8300-2f01184a15d3","Type":"ContainerDied","Data":"c4980b95d79b0465d22191c7fc381400920ea6f1aca149eb74ec1b1a67c535e2"} Oct 30 18:40:56.558361 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.546167 4603 scope.go:122] "RemoveContainer" containerID="f65aac4dd93bea107473a1364b0f53516964b3092ac1cdbb6cfbf7db84119f14" Oct 30 18:40:56.577608 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.577573 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8"] Oct 30 18:40:56.588640 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.588601 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" podUID="c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" containerName="istio-proxy" containerID="cri-o://69363d50945d8c790fa0bd6894e6c1d8e1d28f1e3abc4fc1c5ea7827a69ca489" gracePeriod=30 Oct 30 18:40:56.612911 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.608354 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm"] Oct 30 18:40:56.613367 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.613336 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" podUID="db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" containerName="istio-proxy" containerID="cri-o://0e8affd8ce1ebaa1693512f13563745530ee656c8354929f2932d6fde0ca3fb0" gracePeriod=30 Oct 30 18:40:56.761528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.759231 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d187b51-f5c6-4a63-8300-2f01184a15d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d187b51-f5c6-4a63-8300-2f01184a15d3-istiod-ca-cert\") pod \"1d187b51-f5c6-4a63-8300-2f01184a15d3\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " Oct 30 18:40:56.761528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.759389 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-workload-socket\") pod \"1d187b51-f5c6-4a63-8300-2f01184a15d3\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " Oct 30 18:40:56.761528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.759434 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-data\") pod \"1d187b51-f5c6-4a63-8300-2f01184a15d3\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " Oct 30 18:40:56.761528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.759484 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-credential-socket\") pod \"1d187b51-f5c6-4a63-8300-2f01184a15d3\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " Oct 30 18:40:56.761528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.759558 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-token\" (UniqueName: \"kubernetes.io/projected/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-token\") pod \"1d187b51-f5c6-4a63-8300-2f01184a15d3\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " Oct 30 18:40:56.761528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.759588 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-podinfo\") pod \"1d187b51-f5c6-4a63-8300-2f01184a15d3\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " Oct 30 18:40:56.761528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.759629 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-envoy\") pod \"1d187b51-f5c6-4a63-8300-2f01184a15d3\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " Oct 30 18:40:56.761528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.759704 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d187b51-f5c6-4a63-8300-2f01184a15d3-kube-api-access-xvrsm\" (UniqueName: \"kubernetes.io/projected/1d187b51-f5c6-4a63-8300-2f01184a15d3-kube-api-access-xvrsm\") pod \"1d187b51-f5c6-4a63-8300-2f01184a15d3\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " Oct 30 18:40:56.761528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.759755 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-workload-certs\") pod \"1d187b51-f5c6-4a63-8300-2f01184a15d3\" (UID: \"1d187b51-f5c6-4a63-8300-2f01184a15d3\") " Oct 30 18:40:56.782539 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.779722 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-credential-socket" pod "1d187b51-f5c6-4a63-8300-2f01184a15d3" (UID: "1d187b51-f5c6-4a63-8300-2f01184a15d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:56.805534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.788767 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-workload-certs" pod "1d187b51-f5c6-4a63-8300-2f01184a15d3" (UID: "1d187b51-f5c6-4a63-8300-2f01184a15d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:56.805534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.791052 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-workload-socket" pod "1d187b51-f5c6-4a63-8300-2f01184a15d3" (UID: "1d187b51-f5c6-4a63-8300-2f01184a15d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:56.805534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.804226 4603 generic.go:356] "Generic (PLEG): container finished" podID="d4a3adf6-fb23-4334-8bab-f800b4e8e089" containerID="3fb29526621b83ed763d3401b3b37a008321bb644366f41e7cbc405cadeef19c" exitCode=0 Oct 30 18:40:56.805534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.804310 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" event={"ID":"d4a3adf6-fb23-4334-8bab-f800b4e8e089","Type":"ContainerDied","Data":"3fb29526621b83ed763d3401b3b37a008321bb644366f41e7cbc405cadeef19c"} Oct 30 18:40:56.806968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.806704 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d187b51-f5c6-4a63-8300-2f01184a15d3-istiod-ca-cert" pod "1d187b51-f5c6-4a63-8300-2f01184a15d3" (UID: "1d187b51-f5c6-4a63-8300-2f01184a15d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:56.807843 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.807140 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-data" pod "1d187b51-f5c6-4a63-8300-2f01184a15d3" (UID: "1d187b51-f5c6-4a63-8300-2f01184a15d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:56.846207 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.846124 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:40:56.858472 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.858448 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:40:56.895541 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.870489 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.896432 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.896404 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.897196 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.897169 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d187b51-f5c6-4a63-8300-2f01184a15d3-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.897324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.897201 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.897324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.897219 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.909548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.898236 4603 generic.go:356] "Generic (PLEG): container finished" podID="22e0b280-2b65-4fad-8344-67049190d108" containerID="987b8014d0d4886b3ebdf1c880a7bb694d1169d0fd45406d4d558b7cc98eda45" exitCode=0 Oct 30 18:40:56.909548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.898334 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" event={"ID":"22e0b280-2b65-4fad-8344-67049190d108","Type":"ContainerDied","Data":"987b8014d0d4886b3ebdf1c880a7bb694d1169d0fd45406d4d558b7cc98eda45"} Oct 30 18:40:56.909548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.898389 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" event={"ID":"22e0b280-2b65-4fad-8344-67049190d108","Type":"ContainerDied","Data":"a70932a0e69be2689d6c7f15f49d23a7e99995820976c73873f5b54704807349"} Oct 30 18:40:56.909548 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.898475 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn" Oct 30 18:40:56.991349 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.986642 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-podinfo" pod "1d187b51-f5c6-4a63-8300-2f01184a15d3" (UID: "1d187b51-f5c6-4a63-8300-2f01184a15d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:40:57.000113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.998255 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-podinfo\") pod \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " Oct 30 18:40:57.000113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.998319 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-data\") pod \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " Oct 30 18:40:57.000113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.998408 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-token\" (UniqueName: \"kubernetes.io/projected/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-token\") pod \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " Oct 30 18:40:57.000113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.998451 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-credential-socket\") pod \"b664f4ca-736e-4cb1-991d-985d1e6aff34\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " Oct 30 18:40:57.000113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.998525 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b664f4ca-736e-4cb1-991d-985d1e6aff34-kube-api-access-kqrxs\" (UniqueName: \"kubernetes.io/projected/b664f4ca-736e-4cb1-991d-985d1e6aff34-kube-api-access-kqrxs\") pod \"b664f4ca-736e-4cb1-991d-985d1e6aff34\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " Oct 30 18:40:57.000113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.998598 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-workload-socket\") pod \"b664f4ca-736e-4cb1-991d-985d1e6aff34\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " Oct 30 18:40:57.000113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.998637 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-envoy\") pod \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " Oct 30 18:40:57.000113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.998681 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-envoy\") pod \"b664f4ca-736e-4cb1-991d-985d1e6aff34\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " Oct 30 18:40:57.000113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.998752 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-workload-certs\") pod \"b664f4ca-736e-4cb1-991d-985d1e6aff34\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " Oct 30 18:40:57.000113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.998785 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-credential-socket\") pod \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " Oct 30 18:40:57.000113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.998832 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istiod-ca-cert\") pod \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " Oct 30 18:40:57.000113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.998881 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-workload-socket\") pod \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " Oct 30 18:40:57.000113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.998954 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-podinfo\") pod \"b664f4ca-736e-4cb1-991d-985d1e6aff34\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " Oct 30 18:40:57.000113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.998994 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b664f4ca-736e-4cb1-991d-985d1e6aff34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b664f4ca-736e-4cb1-991d-985d1e6aff34-istiod-ca-cert\") pod \"b664f4ca-736e-4cb1-991d-985d1e6aff34\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " Oct 30 18:40:57.000113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.999028 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-data\") pod \"b664f4ca-736e-4cb1-991d-985d1e6aff34\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " Oct 30 18:40:57.000113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.999068 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-workload-certs\") pod \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " Oct 30 18:40:57.000113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.999121 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-token\" (UniqueName: \"kubernetes.io/projected/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-token\") pod \"b664f4ca-736e-4cb1-991d-985d1e6aff34\" (UID: \"b664f4ca-736e-4cb1-991d-985d1e6aff34\") " Oct 30 18:40:57.000113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.999168 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4a3adf6-fb23-4334-8bab-f800b4e8e089-kube-api-access-ktqqm\" (UniqueName: \"kubernetes.io/projected/d4a3adf6-fb23-4334-8bab-f800b4e8e089-kube-api-access-ktqqm\") pod \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\" (UID: \"d4a3adf6-fb23-4334-8bab-f800b4e8e089\") " Oct 30 18:40:57.000113 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:56.999753 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.003858 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.003829 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-workload-certs" pod "b664f4ca-736e-4cb1-991d-985d1e6aff34" (UID: "b664f4ca-736e-4cb1-991d-985d1e6aff34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:57.006092 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.006065 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-token" pod "1d187b51-f5c6-4a63-8300-2f01184a15d3" (UID: "1d187b51-f5c6-4a63-8300-2f01184a15d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:57.006706 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.006656 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d187b51-f5c6-4a63-8300-2f01184a15d3-kube-api-access-xvrsm" pod "1d187b51-f5c6-4a63-8300-2f01184a15d3" (UID: "1d187b51-f5c6-4a63-8300-2f01184a15d3"). InnerVolumeSpecName "kube-api-access-xvrsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:57.022448 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.022420 4603 scope.go:122] "RemoveContainer" containerID="f65aac4dd93bea107473a1364b0f53516964b3092ac1cdbb6cfbf7db84119f14" Oct 30 18:40:57.025791 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.025759 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-workload-socket" pod "b664f4ca-736e-4cb1-991d-985d1e6aff34" (UID: "b664f4ca-736e-4cb1-991d-985d1e6aff34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:57.026002 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.025972 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-data" pod "b664f4ca-736e-4cb1-991d-985d1e6aff34" (UID: "b664f4ca-736e-4cb1-991d-985d1e6aff34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:57.028949 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:40:57.027866 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f65aac4dd93bea107473a1364b0f53516964b3092ac1cdbb6cfbf7db84119f14\": container with ID starting with f65aac4dd93bea107473a1364b0f53516964b3092ac1cdbb6cfbf7db84119f14 not found: ID does not exist" containerID="f65aac4dd93bea107473a1364b0f53516964b3092ac1cdbb6cfbf7db84119f14" Oct 30 18:40:57.028949 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.027908 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f65aac4dd93bea107473a1364b0f53516964b3092ac1cdbb6cfbf7db84119f14"} err="failed to get container status \"f65aac4dd93bea107473a1364b0f53516964b3092ac1cdbb6cfbf7db84119f14\": rpc error: code = NotFound desc = could not find container \"f65aac4dd93bea107473a1364b0f53516964b3092ac1cdbb6cfbf7db84119f14\": container with ID starting with f65aac4dd93bea107473a1364b0f53516964b3092ac1cdbb6cfbf7db84119f14 not found: ID does not exist" Oct 30 18:40:57.028949 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.027936 4603 scope.go:122] "RemoveContainer" containerID="987b8014d0d4886b3ebdf1c880a7bb694d1169d0fd45406d4d558b7cc98eda45" Oct 30 18:40:57.030652 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.030577 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-data" pod "d4a3adf6-fb23-4334-8bab-f800b4e8e089" (UID: "d4a3adf6-fb23-4334-8bab-f800b4e8e089"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:57.031043 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.031013 4603 generic.go:356] "Generic (PLEG): container finished" podID="b664f4ca-736e-4cb1-991d-985d1e6aff34" containerID="80d5343a4d7fab6f7441d8100515aaec7fabac590594da9544683a876a70c15d" exitCode=0 Oct 30 18:40:57.031148 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.031101 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" event={"ID":"b664f4ca-736e-4cb1-991d-985d1e6aff34","Type":"ContainerDied","Data":"80d5343a4d7fab6f7441d8100515aaec7fabac590594da9544683a876a70c15d"} Oct 30 18:40:57.031215 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.031202 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc" Oct 30 18:40:57.033670 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.033639 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-envoy" pod "1d187b51-f5c6-4a63-8300-2f01184a15d3" (UID: "1d187b51-f5c6-4a63-8300-2f01184a15d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:57.040528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.035619 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-workload-certs" pod "d4a3adf6-fb23-4334-8bab-f800b4e8e089" (UID: "d4a3adf6-fb23-4334-8bab-f800b4e8e089"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:57.040528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.039578 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-workload-socket" pod "d4a3adf6-fb23-4334-8bab-f800b4e8e089" (UID: "d4a3adf6-fb23-4334-8bab-f800b4e8e089"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:57.040528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.039697 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4a3adf6-fb23-4334-8bab-f800b4e8e089-kube-api-access-ktqqm" pod "d4a3adf6-fb23-4334-8bab-f800b4e8e089" (UID: "d4a3adf6-fb23-4334-8bab-f800b4e8e089"). InnerVolumeSpecName "kube-api-access-ktqqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:57.042793 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.041228 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf" podUID="ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:57.042793 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.041589 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-credential-socket" pod "d4a3adf6-fb23-4334-8bab-f800b4e8e089" (UID: "d4a3adf6-fb23-4334-8bab-f800b4e8e089"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:57.042793 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.042521 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b664f4ca-736e-4cb1-991d-985d1e6aff34-istiod-ca-cert" pod "b664f4ca-736e-4cb1-991d-985d1e6aff34" (UID: "b664f4ca-736e-4cb1-991d-985d1e6aff34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:57.043487 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.043444 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-credential-socket" pod "b664f4ca-736e-4cb1-991d-985d1e6aff34" (UID: "b664f4ca-736e-4cb1-991d-985d1e6aff34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:57.043811 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.043777 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istiod-ca-cert" pod "d4a3adf6-fb23-4334-8bab-f800b4e8e089" (UID: "d4a3adf6-fb23-4334-8bab-f800b4e8e089"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:57.138624 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.138532 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.138855 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.138834 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.138967 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.138953 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.139095 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.139073 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.139200 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.139187 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.139303 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.139290 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.139412 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.139399 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.139547 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.139531 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d187b51-f5c6-4a63-8300-2f01184a15d3-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.139667 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.139654 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b664f4ca-736e-4cb1-991d-985d1e6aff34-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.139761 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.139749 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.139850 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.139839 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.139957 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.139945 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvrsm\" (UniqueName: \"kubernetes.io/projected/1d187b51-f5c6-4a63-8300-2f01184a15d3-kube-api-access-xvrsm\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.140052 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.140039 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktqqm\" (UniqueName: \"kubernetes.io/projected/d4a3adf6-fb23-4334-8bab-f800b4e8e089-kube-api-access-ktqqm\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.140144 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.140133 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.140381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.140361 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-token" pod "b664f4ca-736e-4cb1-991d-985d1e6aff34" (UID: "b664f4ca-736e-4cb1-991d-985d1e6aff34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:57.168526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.167705 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-podinfo" pod "d4a3adf6-fb23-4334-8bab-f800b4e8e089" (UID: "d4a3adf6-fb23-4334-8bab-f800b4e8e089"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:40:57.168526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.167837 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b664f4ca-736e-4cb1-991d-985d1e6aff34-kube-api-access-kqrxs" pod "b664f4ca-736e-4cb1-991d-985d1e6aff34" (UID: "b664f4ca-736e-4cb1-991d-985d1e6aff34"). InnerVolumeSpecName "kube-api-access-kqrxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:57.168526 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.168481 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-token" pod "d4a3adf6-fb23-4334-8bab-f800b4e8e089" (UID: "d4a3adf6-fb23-4334-8bab-f800b4e8e089"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:57.169775 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.169654 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-podinfo" pod "b664f4ca-736e-4cb1-991d-985d1e6aff34" (UID: "b664f4ca-736e-4cb1-991d-985d1e6aff34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:40:57.178224 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.178181 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-envoy" pod "b664f4ca-736e-4cb1-991d-985d1e6aff34" (UID: "b664f4ca-736e-4cb1-991d-985d1e6aff34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:57.197370 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.192105 4603 scope.go:122] "RemoveContainer" containerID="16d8aa22cecbcf7f4440eecd90d2bc6e688028c1e367f0d43f4f2c01b4726a68" Oct 30 18:40:57.216063 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.216030 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-envoy" pod "d4a3adf6-fb23-4334-8bab-f800b4e8e089" (UID: "d4a3adf6-fb23-4334-8bab-f800b4e8e089"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:57.223665 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.223303 4603 scope.go:122] "RemoveContainer" containerID="987b8014d0d4886b3ebdf1c880a7bb694d1169d0fd45406d4d558b7cc98eda45" Oct 30 18:40:57.224232 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:40:57.223900 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"987b8014d0d4886b3ebdf1c880a7bb694d1169d0fd45406d4d558b7cc98eda45\": container with ID starting with 987b8014d0d4886b3ebdf1c880a7bb694d1169d0fd45406d4d558b7cc98eda45 not found: ID does not exist" containerID="987b8014d0d4886b3ebdf1c880a7bb694d1169d0fd45406d4d558b7cc98eda45" Oct 30 18:40:57.224232 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.223942 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"987b8014d0d4886b3ebdf1c880a7bb694d1169d0fd45406d4d558b7cc98eda45"} err="failed to get container status \"987b8014d0d4886b3ebdf1c880a7bb694d1169d0fd45406d4d558b7cc98eda45\": rpc error: code = NotFound desc = could not find container \"987b8014d0d4886b3ebdf1c880a7bb694d1169d0fd45406d4d558b7cc98eda45\": container with ID starting with 987b8014d0d4886b3ebdf1c880a7bb694d1169d0fd45406d4d558b7cc98eda45 not found: ID does not exist" Oct 30 18:40:57.224232 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.223970 4603 scope.go:122] "RemoveContainer" containerID="16d8aa22cecbcf7f4440eecd90d2bc6e688028c1e367f0d43f4f2c01b4726a68" Oct 30 18:40:57.225339 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:40:57.224952 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d8aa22cecbcf7f4440eecd90d2bc6e688028c1e367f0d43f4f2c01b4726a68\": container with ID starting with 16d8aa22cecbcf7f4440eecd90d2bc6e688028c1e367f0d43f4f2c01b4726a68 not found: ID does not exist" containerID="16d8aa22cecbcf7f4440eecd90d2bc6e688028c1e367f0d43f4f2c01b4726a68" Oct 30 18:40:57.225339 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.224988 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d8aa22cecbcf7f4440eecd90d2bc6e688028c1e367f0d43f4f2c01b4726a68"} err="failed to get container status \"16d8aa22cecbcf7f4440eecd90d2bc6e688028c1e367f0d43f4f2c01b4726a68\": rpc error: code = NotFound desc = could not find container \"16d8aa22cecbcf7f4440eecd90d2bc6e688028c1e367f0d43f4f2c01b4726a68\": container with ID starting with 16d8aa22cecbcf7f4440eecd90d2bc6e688028c1e367f0d43f4f2c01b4726a68 not found: ID does not exist" Oct 30 18:40:57.225339 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.225012 4603 scope.go:122] "RemoveContainer" containerID="80d5343a4d7fab6f7441d8100515aaec7fabac590594da9544683a876a70c15d" Oct 30 18:40:57.271530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.254014 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.273697 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.273679 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.274055 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.274034 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.274370 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.274352 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b664f4ca-736e-4cb1-991d-985d1e6aff34-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.274633 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.274619 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.274980 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.274865 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4a3adf6-fb23-4334-8bab-f800b4e8e089-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.275228 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.275211 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqrxs\" (UniqueName: \"kubernetes.io/projected/b664f4ca-736e-4cb1-991d-985d1e6aff34-kube-api-access-kqrxs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.309957 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.309912 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn"] Oct 30 18:40:57.327349 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.327316 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-4h9sn"] Oct 30 18:40:57.462613 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.450588 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:57.502136 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.502101 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22e0b280-2b65-4fad-8344-67049190d108" path="/var/lib/kubelet/pods/22e0b280-2b65-4fad-8344-67049190d108/volumes" Oct 30 18:40:57.539732 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.539699 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc"] Oct 30 18:40:57.593048 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:57.558428 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-2n6sc"] Oct 30 18:40:58.037870 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:58.037840 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" Oct 30 18:40:58.038376 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:58.037835 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v" event={"ID":"d4a3adf6-fb23-4334-8bab-f800b4e8e089","Type":"ContainerDied","Data":"f6ae81debcf20b88835927277b634002f56de72cc703590641a84677bcab4bad"} Oct 30 18:40:58.038376 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:58.037997 4603 scope.go:122] "RemoveContainer" containerID="3fb29526621b83ed763d3401b3b37a008321bb644366f41e7cbc405cadeef19c" Oct 30 18:40:58.041314 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:58.041295 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x" Oct 30 18:40:58.088900 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:58.088874 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v"] Oct 30 18:40:58.100599 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:58.100575 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-mtv5v"] Oct 30 18:40:58.127685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:58.127661 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x"] Oct 30 18:40:58.135358 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:58.135338 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-msq9x"] Oct 30 18:40:59.152219 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:59.152185 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:59.408592 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:59.408491 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d187b51-f5c6-4a63-8300-2f01184a15d3" path="/var/lib/kubelet/pods/1d187b51-f5c6-4a63-8300-2f01184a15d3/volumes" Oct 30 18:40:59.409066 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:59.409043 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b664f4ca-736e-4cb1-991d-985d1e6aff34" path="/var/lib/kubelet/pods/b664f4ca-736e-4cb1-991d-985d1e6aff34/volumes" Oct 30 18:40:59.409570 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:40:59.409549 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4a3adf6-fb23-4334-8bab-f800b4e8e089" path="/var/lib/kubelet/pods/d4a3adf6-fb23-4334-8bab-f800b4e8e089/volumes" Oct 30 18:41:00.705690 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:00.705658 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:00.853865 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:00.853816 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:01.048826 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.048790 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:01.564981 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.564948 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:01.704577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.704515 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:41:01.747473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.746533 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-workload-certs\") pod \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " Oct 30 18:41:01.747473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.746621 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-workload-socket\") pod \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " Oct 30 18:41:01.747473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.746657 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-token\" (UniqueName: \"kubernetes.io/projected/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-token\") pod \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " Oct 30 18:41:01.747473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.746706 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-podinfo\") pod \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " Oct 30 18:41:01.747473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.746747 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-data\") pod \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " Oct 30 18:41:01.747473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.746836 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-credential-socket\") pod \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " Oct 30 18:41:01.747473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.746891 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-envoy\") pod \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " Oct 30 18:41:01.747473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.746975 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istiod-ca-cert\") pod \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " Oct 30 18:41:01.747473 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.747011 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-kube-api-access-scvgj\" (UniqueName: \"kubernetes.io/projected/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-kube-api-access-scvgj\") pod \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\" (UID: \"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64\") " Oct 30 18:41:01.748342 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.747550 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-workload-certs" pod "47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" (UID: "47cefcfc-ddae-4c02-ad2c-ae2cb8769e64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:01.748342 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.747663 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-credential-socket" pod "47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" (UID: "47cefcfc-ddae-4c02-ad2c-ae2cb8769e64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:01.748342 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.747984 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-workload-socket" pod "47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" (UID: "47cefcfc-ddae-4c02-ad2c-ae2cb8769e64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:01.748342 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.748025 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istiod-ca-cert" pod "47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" (UID: "47cefcfc-ddae-4c02-ad2c-ae2cb8769e64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:01.748342 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.748290 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-data" pod "47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" (UID: "47cefcfc-ddae-4c02-ad2c-ae2cb8769e64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:01.753989 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.753961 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-token" pod "47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" (UID: "47cefcfc-ddae-4c02-ad2c-ae2cb8769e64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:01.754137 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.754070 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-kube-api-access-scvgj" pod "47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" (UID: "47cefcfc-ddae-4c02-ad2c-ae2cb8769e64"). InnerVolumeSpecName "kube-api-access-scvgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:01.754212 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.754141 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-podinfo" pod "47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" (UID: "47cefcfc-ddae-4c02-ad2c-ae2cb8769e64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:41:01.754212 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.754154 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-envoy" pod "47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" (UID: "47cefcfc-ddae-4c02-ad2c-ae2cb8769e64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:01.848602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.848461 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:01.848602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.848491 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:01.848602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.848519 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:01.848602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.848532 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scvgj\" (UniqueName: \"kubernetes.io/projected/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-kube-api-access-scvgj\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:01.848602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.848542 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:01.848602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.848551 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:01.848602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.848559 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:01.848602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.848567 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:01.848602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.848574 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:01.882457 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.882430 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:41:01.945734 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.945708 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:41:01.949834 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.949810 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-envoy\") pod \"d5567046-aa24-421a-8339-fdd9cf56e62c\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " Oct 30 18:41:01.949899 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.949861 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-podinfo\") pod \"d5567046-aa24-421a-8339-fdd9cf56e62c\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " Oct 30 18:41:01.949948 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.949909 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-token\" (UniqueName: \"kubernetes.io/projected/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-token\") pod \"d5567046-aa24-421a-8339-fdd9cf56e62c\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " Oct 30 18:41:01.950000 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.949983 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5567046-aa24-421a-8339-fdd9cf56e62c-kube-api-access-4gmmt\" (UniqueName: \"kubernetes.io/projected/d5567046-aa24-421a-8339-fdd9cf56e62c-kube-api-access-4gmmt\") pod \"d5567046-aa24-421a-8339-fdd9cf56e62c\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " Oct 30 18:41:01.950142 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.950108 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-credential-socket\") pod \"d5567046-aa24-421a-8339-fdd9cf56e62c\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " Oct 30 18:41:01.950223 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.950144 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-workload-certs\") pod \"d5567046-aa24-421a-8339-fdd9cf56e62c\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " Oct 30 18:41:01.950296 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.950240 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-data\") pod \"d5567046-aa24-421a-8339-fdd9cf56e62c\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " Oct 30 18:41:01.950296 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.950284 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5567046-aa24-421a-8339-fdd9cf56e62c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5567046-aa24-421a-8339-fdd9cf56e62c-istiod-ca-cert\") pod \"d5567046-aa24-421a-8339-fdd9cf56e62c\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " Oct 30 18:41:01.950412 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.950318 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-workload-socket\") pod \"d5567046-aa24-421a-8339-fdd9cf56e62c\" (UID: \"d5567046-aa24-421a-8339-fdd9cf56e62c\") " Oct 30 18:41:01.950486 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.950407 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-credential-socket" pod "d5567046-aa24-421a-8339-fdd9cf56e62c" (UID: "d5567046-aa24-421a-8339-fdd9cf56e62c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:01.950715 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.950661 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-workload-socket" pod "d5567046-aa24-421a-8339-fdd9cf56e62c" (UID: "d5567046-aa24-421a-8339-fdd9cf56e62c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:01.950715 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.950699 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-data" pod "d5567046-aa24-421a-8339-fdd9cf56e62c" (UID: "d5567046-aa24-421a-8339-fdd9cf56e62c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:01.950959 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.950775 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-workload-certs" pod "d5567046-aa24-421a-8339-fdd9cf56e62c" (UID: "d5567046-aa24-421a-8339-fdd9cf56e62c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:01.951021 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.950989 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5567046-aa24-421a-8339-fdd9cf56e62c-istiod-ca-cert" pod "d5567046-aa24-421a-8339-fdd9cf56e62c" (UID: "d5567046-aa24-421a-8339-fdd9cf56e62c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:01.951084 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.951064 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:01.951151 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.951086 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:01.951151 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.951102 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:01.951151 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.951117 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:01.952374 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.952350 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-token" pod "d5567046-aa24-421a-8339-fdd9cf56e62c" (UID: "d5567046-aa24-421a-8339-fdd9cf56e62c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:01.952689 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.952667 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-podinfo" pod "d5567046-aa24-421a-8339-fdd9cf56e62c" (UID: "d5567046-aa24-421a-8339-fdd9cf56e62c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:41:01.953418 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.953374 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5567046-aa24-421a-8339-fdd9cf56e62c-kube-api-access-4gmmt" pod "d5567046-aa24-421a-8339-fdd9cf56e62c" (UID: "d5567046-aa24-421a-8339-fdd9cf56e62c"). InnerVolumeSpecName "kube-api-access-4gmmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:01.953533 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.953468 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-envoy" pod "d5567046-aa24-421a-8339-fdd9cf56e62c" (UID: "d5567046-aa24-421a-8339-fdd9cf56e62c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:01.967184 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:01.967163 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:41:02.051893 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.051853 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-workload-socket\") pod \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " Oct 30 18:41:02.052084 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.051902 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-workload-certs\") pod \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " Oct 30 18:41:02.052084 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.051946 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-token\" (UniqueName: \"kubernetes.io/projected/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-token\") pod \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " Oct 30 18:41:02.052084 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.051981 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-workload-certs\") pod \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " Oct 30 18:41:02.052084 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.052041 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-data\") pod \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " Oct 30 18:41:02.052324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.052169 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-token\" (UniqueName: \"kubernetes.io/projected/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-token\") pod \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " Oct 30 18:41:02.052324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.052220 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-envoy\") pod \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " Oct 30 18:41:02.052324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.052254 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-kube-api-access-5lfq6\" (UniqueName: \"kubernetes.io/projected/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-kube-api-access-5lfq6\") pod \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " Oct 30 18:41:02.052324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.052173 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-workload-socket" pod "c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" (UID: "c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.052324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.052287 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-workload-socket\") pod \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " Oct 30 18:41:02.052324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.052320 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-credential-socket\") pod \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " Oct 30 18:41:02.052684 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.052367 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istiod-ca-cert\") pod \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " Oct 30 18:41:02.052684 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.052386 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-workload-certs" pod "db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" (UID: "db8b0a7a-6aef-4237-acb3-c9e2ee5112fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.052684 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.052394 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istiod-ca-cert\") pod \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " Oct 30 18:41:02.052684 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.052423 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-data" pod "db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" (UID: "db8b0a7a-6aef-4237-acb3-c9e2ee5112fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.052684 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.052453 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-data\") pod \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " Oct 30 18:41:02.052684 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.052484 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-podinfo\") pod \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " Oct 30 18:41:02.052684 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.052555 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-kube-api-access-264ph\" (UniqueName: \"kubernetes.io/projected/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-kube-api-access-264ph\") pod \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " Oct 30 18:41:02.053058 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.052691 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-envoy\") pod \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\" (UID: \"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd\") " Oct 30 18:41:02.053058 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.052702 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-credential-socket" pod "db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" (UID: "db8b0a7a-6aef-4237-acb3-c9e2ee5112fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.053058 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.052738 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-podinfo\") pod \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " Oct 30 18:41:02.053058 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.052778 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-credential-socket\") pod \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\" (UID: \"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8\") " Oct 30 18:41:02.053058 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.052786 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istiod-ca-cert" pod "db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" (UID: "db8b0a7a-6aef-4237-acb3-c9e2ee5112fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:02.053058 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.052894 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-workload-socket" pod "db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" (UID: "db8b0a7a-6aef-4237-acb3-c9e2ee5112fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.053580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.053101 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-credential-socket" pod "c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" (UID: "c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.053580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.053182 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-workload-certs" pod "c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" (UID: "c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.053580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.053231 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istiod-ca-cert" pod "c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" (UID: "c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:02.053580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.053321 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5567046-aa24-421a-8339-fdd9cf56e62c-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.053580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.053342 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.053580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.053356 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.053580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.053357 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-data" pod "c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" (UID: "c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.053580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.053372 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.053580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.053388 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.053580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.053405 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.053580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.053419 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5567046-aa24-421a-8339-fdd9cf56e62c-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.053580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.053433 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.053580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.053448 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gmmt\" (UniqueName: \"kubernetes.io/projected/d5567046-aa24-421a-8339-fdd9cf56e62c-kube-api-access-4gmmt\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.053580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.053462 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.053580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.053476 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.053580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.053490 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.053580 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.053542 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.055198 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.055174 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-token" pod "c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" (UID: "c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:02.055592 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.055571 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-token" pod "db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" (UID: "db8b0a7a-6aef-4237-acb3-c9e2ee5112fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:02.055945 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.055927 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-envoy" pod "db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" (UID: "db8b0a7a-6aef-4237-acb3-c9e2ee5112fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.056141 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.056119 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-kube-api-access-264ph" pod "c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" (UID: "c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8"). InnerVolumeSpecName "kube-api-access-264ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:02.056230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.056208 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-podinfo" pod "db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" (UID: "db8b0a7a-6aef-4237-acb3-c9e2ee5112fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:41:02.056905 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.056887 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-envoy" pod "c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" (UID: "c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.057346 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.057328 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-kube-api-access-5lfq6" pod "db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" (UID: "db8b0a7a-6aef-4237-acb3-c9e2ee5112fd"). InnerVolumeSpecName "kube-api-access-5lfq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:02.057437 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.057419 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-podinfo" pod "c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" (UID: "c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:41:02.067077 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.067054 4603 generic.go:356] "Generic (PLEG): container finished" podID="d5567046-aa24-421a-8339-fdd9cf56e62c" containerID="ac2d2ba01f24fe73bc3985faf348fec2383e49bb1f192b3800f6546e07025dfc" exitCode=0 Oct 30 18:41:02.067175 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.067102 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" Oct 30 18:41:02.067175 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.067141 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" event={"ID":"d5567046-aa24-421a-8339-fdd9cf56e62c","Type":"ContainerDied","Data":"ac2d2ba01f24fe73bc3985faf348fec2383e49bb1f192b3800f6546e07025dfc"} Oct 30 18:41:02.067268 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.067198 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc" event={"ID":"d5567046-aa24-421a-8339-fdd9cf56e62c","Type":"ContainerDied","Data":"9f278f67587af701c1136ee6b46575434ef728bf9cb94fa8a43e555ff5979637"} Oct 30 18:41:02.067268 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.067225 4603 scope.go:122] "RemoveContainer" containerID="ac2d2ba01f24fe73bc3985faf348fec2383e49bb1f192b3800f6546e07025dfc" Oct 30 18:41:02.068688 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.068666 4603 generic.go:356] "Generic (PLEG): container finished" podID="db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" containerID="0e8affd8ce1ebaa1693512f13563745530ee656c8354929f2932d6fde0ca3fb0" exitCode=0 Oct 30 18:41:02.069006 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.068694 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" event={"ID":"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd","Type":"ContainerDied","Data":"0e8affd8ce1ebaa1693512f13563745530ee656c8354929f2932d6fde0ca3fb0"} Oct 30 18:41:02.069006 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.068732 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" Oct 30 18:41:02.069006 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.068739 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm" event={"ID":"db8b0a7a-6aef-4237-acb3-c9e2ee5112fd","Type":"ContainerDied","Data":"361fbc1b708fddbe1b9cabbd02f27882ca2c97b437c805fe699b8e0408484e60"} Oct 30 18:41:02.074821 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.074798 4603 generic.go:356] "Generic (PLEG): container finished" podID="47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" containerID="88a9eaba348c52a304694c657c04c62b48cd9d534a7106e34e43a3fe117a1905" exitCode=0 Oct 30 18:41:02.074999 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.074866 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" Oct 30 18:41:02.074999 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.074918 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" event={"ID":"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64","Type":"ContainerDied","Data":"88a9eaba348c52a304694c657c04c62b48cd9d534a7106e34e43a3fe117a1905"} Oct 30 18:41:02.074999 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.074960 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw" event={"ID":"47cefcfc-ddae-4c02-ad2c-ae2cb8769e64","Type":"ContainerDied","Data":"26a5a98cefa4f2f963d18e548596e3471768c80e1b172378b5ec1618205b3059"} Oct 30 18:41:02.076779 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.076727 4603 generic.go:356] "Generic (PLEG): container finished" podID="c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" containerID="69363d50945d8c790fa0bd6894e6c1d8e1d28f1e3abc4fc1c5ea7827a69ca489" exitCode=0 Oct 30 18:41:02.076877 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.076792 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" Oct 30 18:41:02.076877 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.076760 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" event={"ID":"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8","Type":"ContainerDied","Data":"69363d50945d8c790fa0bd6894e6c1d8e1d28f1e3abc4fc1c5ea7827a69ca489"} Oct 30 18:41:02.076983 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.076882 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8" event={"ID":"c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8","Type":"ContainerDied","Data":"ddd2a011a67431dbde720c2ed8fad2e78e86a73bc2a10b0e431f916d47f4cf9f"} Oct 30 18:41:02.078135 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.078118 4603 scope.go:122] "RemoveContainer" containerID="ac2d2ba01f24fe73bc3985faf348fec2383e49bb1f192b3800f6546e07025dfc" Oct 30 18:41:02.078542 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:41:02.078480 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac2d2ba01f24fe73bc3985faf348fec2383e49bb1f192b3800f6546e07025dfc\": container with ID starting with ac2d2ba01f24fe73bc3985faf348fec2383e49bb1f192b3800f6546e07025dfc not found: ID does not exist" containerID="ac2d2ba01f24fe73bc3985faf348fec2383e49bb1f192b3800f6546e07025dfc" Oct 30 18:41:02.078639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.078552 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac2d2ba01f24fe73bc3985faf348fec2383e49bb1f192b3800f6546e07025dfc"} err="failed to get container status \"ac2d2ba01f24fe73bc3985faf348fec2383e49bb1f192b3800f6546e07025dfc\": rpc error: code = NotFound desc = could not find container \"ac2d2ba01f24fe73bc3985faf348fec2383e49bb1f192b3800f6546e07025dfc\": container with ID starting with ac2d2ba01f24fe73bc3985faf348fec2383e49bb1f192b3800f6546e07025dfc not found: ID does not exist" Oct 30 18:41:02.078639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.078576 4603 scope.go:122] "RemoveContainer" containerID="0e8affd8ce1ebaa1693512f13563745530ee656c8354929f2932d6fde0ca3fb0" Oct 30 18:41:02.089825 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.089806 4603 scope.go:122] "RemoveContainer" containerID="0e8affd8ce1ebaa1693512f13563745530ee656c8354929f2932d6fde0ca3fb0" Oct 30 18:41:02.090191 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:41:02.090157 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e8affd8ce1ebaa1693512f13563745530ee656c8354929f2932d6fde0ca3fb0\": container with ID starting with 0e8affd8ce1ebaa1693512f13563745530ee656c8354929f2932d6fde0ca3fb0 not found: ID does not exist" containerID="0e8affd8ce1ebaa1693512f13563745530ee656c8354929f2932d6fde0ca3fb0" Oct 30 18:41:02.090271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.090201 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e8affd8ce1ebaa1693512f13563745530ee656c8354929f2932d6fde0ca3fb0"} err="failed to get container status \"0e8affd8ce1ebaa1693512f13563745530ee656c8354929f2932d6fde0ca3fb0\": rpc error: code = NotFound desc = could not find container \"0e8affd8ce1ebaa1693512f13563745530ee656c8354929f2932d6fde0ca3fb0\": container with ID starting with 0e8affd8ce1ebaa1693512f13563745530ee656c8354929f2932d6fde0ca3fb0 not found: ID does not exist" Oct 30 18:41:02.090271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.090223 4603 scope.go:122] "RemoveContainer" containerID="88a9eaba348c52a304694c657c04c62b48cd9d534a7106e34e43a3fe117a1905" Oct 30 18:41:02.100159 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.100141 4603 scope.go:122] "RemoveContainer" containerID="88a9eaba348c52a304694c657c04c62b48cd9d534a7106e34e43a3fe117a1905" Oct 30 18:41:02.100667 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:41:02.100639 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88a9eaba348c52a304694c657c04c62b48cd9d534a7106e34e43a3fe117a1905\": container with ID starting with 88a9eaba348c52a304694c657c04c62b48cd9d534a7106e34e43a3fe117a1905 not found: ID does not exist" containerID="88a9eaba348c52a304694c657c04c62b48cd9d534a7106e34e43a3fe117a1905" Oct 30 18:41:02.100771 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.100677 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a9eaba348c52a304694c657c04c62b48cd9d534a7106e34e43a3fe117a1905"} err="failed to get container status \"88a9eaba348c52a304694c657c04c62b48cd9d534a7106e34e43a3fe117a1905\": rpc error: code = NotFound desc = could not find container \"88a9eaba348c52a304694c657c04c62b48cd9d534a7106e34e43a3fe117a1905\": container with ID starting with 88a9eaba348c52a304694c657c04c62b48cd9d534a7106e34e43a3fe117a1905 not found: ID does not exist" Oct 30 18:41:02.100771 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.100699 4603 scope.go:122] "RemoveContainer" containerID="69363d50945d8c790fa0bd6894e6c1d8e1d28f1e3abc4fc1c5ea7827a69ca489" Oct 30 18:41:02.109561 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.109538 4603 scope.go:122] "RemoveContainer" containerID="69363d50945d8c790fa0bd6894e6c1d8e1d28f1e3abc4fc1c5ea7827a69ca489" Oct 30 18:41:02.109823 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:41:02.109805 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69363d50945d8c790fa0bd6894e6c1d8e1d28f1e3abc4fc1c5ea7827a69ca489\": container with ID starting with 69363d50945d8c790fa0bd6894e6c1d8e1d28f1e3abc4fc1c5ea7827a69ca489 not found: ID does not exist" containerID="69363d50945d8c790fa0bd6894e6c1d8e1d28f1e3abc4fc1c5ea7827a69ca489" Oct 30 18:41:02.109897 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.109829 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69363d50945d8c790fa0bd6894e6c1d8e1d28f1e3abc4fc1c5ea7827a69ca489"} err="failed to get container status \"69363d50945d8c790fa0bd6894e6c1d8e1d28f1e3abc4fc1c5ea7827a69ca489\": rpc error: code = NotFound desc = could not find container \"69363d50945d8c790fa0bd6894e6c1d8e1d28f1e3abc4fc1c5ea7827a69ca489\": container with ID starting with 69363d50945d8c790fa0bd6894e6c1d8e1d28f1e3abc4fc1c5ea7827a69ca489 not found: ID does not exist" Oct 30 18:41:02.126562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.126458 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc"] Oct 30 18:41:02.129129 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.128862 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-68cqc"] Oct 30 18:41:02.154098 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.154067 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.154098 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.154093 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.154098 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.154103 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-264ph\" (UniqueName: \"kubernetes.io/projected/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-kube-api-access-264ph\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.154358 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.154113 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.154358 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.154122 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.154358 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.154129 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.154358 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.154136 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.154358 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.154143 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.154358 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.154151 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lfq6\" (UniqueName: \"kubernetes.io/projected/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd-kube-api-access-5lfq6\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.154358 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.154159 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.215274 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.215183 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm"] Oct 30 18:41:02.219124 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.219102 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-pmxjm"] Oct 30 18:41:02.252466 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.252425 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw"] Oct 30 18:41:02.256566 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.256541 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-mgrkw"] Oct 30 18:41:02.290261 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.290239 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8"] Oct 30 18:41:02.296310 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.296291 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-gv6s8"] Oct 30 18:41:02.749231 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:02.749186 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:03.232198 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:03.232165 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:03.409049 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:03.409019 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" path="/var/lib/kubelet/pods/47cefcfc-ddae-4c02-ad2c-ae2cb8769e64/volumes" Oct 30 18:41:03.409464 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:03.409450 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" path="/var/lib/kubelet/pods/c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8/volumes" Oct 30 18:41:03.409879 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:03.409865 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5567046-aa24-421a-8339-fdd9cf56e62c" path="/var/lib/kubelet/pods/d5567046-aa24-421a-8339-fdd9cf56e62c/volumes" Oct 30 18:41:03.410600 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:03.410586 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" path="/var/lib/kubelet/pods/db8b0a7a-6aef-4237-acb3-c9e2ee5112fd/volumes" Oct 30 18:41:04.450480 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:04.450451 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:06.040024 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:06.039984 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:07.839305 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:07.839269 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:08.071465 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:08.071426 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:09.451625 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:09.451589 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:10.877053 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:10.877018 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r"] Oct 30 18:41:10.910760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:10.910731 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw"] Oct 30 18:41:11.063495 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:11.063457 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:11.203253 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:11.203165 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:12.793050 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:12.793005 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:14.543964 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:14.543928 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:15.877313 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:15.877276 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r" podUID="f7d52a97-9fde-4a62-95ab-b6fa18c65db9" containerName="payload-processing" containerID="cri-o://e8f13b5998bd6d3ce2137a2325229c80dcb3337b6d1480a2fdc3174bb532a65c" gracePeriod=25 Oct 30 18:41:15.911241 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:15.911186 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw" podUID="4d9f4fc8-48f9-4e12-afa4-fecea8575b5c" containerName="payload-pre-processing" containerID="cri-o://32bdcf8d25dd9ebc0df48beb079f3030d10113d3c0ef604e737cdcec61987f6d" gracePeriod=25 Oct 30 18:41:16.264822 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:16.264562 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:17.948492 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:17.948449 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:18.754074 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:18.754035 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:19.483043 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:19.483014 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:19.667986 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:19.662200 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:21.247112 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:21.247073 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:21.789490 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:21.789453 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw" podUID="4d9f4fc8-48f9-4e12-afa4-fecea8575b5c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:41:21.789490 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:21.789466 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r" podUID="f7d52a97-9fde-4a62-95ab-b6fa18c65db9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:41:21.929818 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:41:21.929780 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:41:22.845883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:22.845850 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:24.486291 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:24.486264 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:26.042685 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:26.042650 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:27.640902 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:27.640867 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:29.237630 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:29.237602 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:30.852594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:30.852556 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:31.789739 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:31.789704 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r" podUID="f7d52a97-9fde-4a62-95ab-b6fa18c65db9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:41:31.789925 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:31.789739 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw" podUID="4d9f4fc8-48f9-4e12-afa4-fecea8575b5c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:41:32.444405 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:32.444370 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:33.799183 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:33.799142 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:34.333840 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:34.333804 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:34.468452 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:34.468398 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:35.096881 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:35.096847 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:36.111907 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.111872 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:36.190517 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.190478 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r" Oct 30 18:41:36.211422 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.211401 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw" Oct 30 18:41:36.260512 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.260483 4603 generic.go:356] "Generic (PLEG): container finished" podID="4d9f4fc8-48f9-4e12-afa4-fecea8575b5c" containerID="32bdcf8d25dd9ebc0df48beb079f3030d10113d3c0ef604e737cdcec61987f6d" exitCode=0 Oct 30 18:41:36.260655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.260537 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw" event={"ID":"4d9f4fc8-48f9-4e12-afa4-fecea8575b5c","Type":"ContainerDied","Data":"32bdcf8d25dd9ebc0df48beb079f3030d10113d3c0ef604e737cdcec61987f6d"} Oct 30 18:41:36.260655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.260591 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw" event={"ID":"4d9f4fc8-48f9-4e12-afa4-fecea8575b5c","Type":"ContainerDied","Data":"e77aa3ed21e827dde349f3ab4dca7f5cb536d665786965a2e6df8d2c3bd47090"} Oct 30 18:41:36.260655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.260551 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw" Oct 30 18:41:36.260655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.260621 4603 scope.go:122] "RemoveContainer" containerID="32bdcf8d25dd9ebc0df48beb079f3030d10113d3c0ef604e737cdcec61987f6d" Oct 30 18:41:36.261975 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.261954 4603 generic.go:356] "Generic (PLEG): container finished" podID="f7d52a97-9fde-4a62-95ab-b6fa18c65db9" containerID="e8f13b5998bd6d3ce2137a2325229c80dcb3337b6d1480a2fdc3174bb532a65c" exitCode=0 Oct 30 18:41:36.262110 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.261999 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r" Oct 30 18:41:36.262110 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.262038 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r" event={"ID":"f7d52a97-9fde-4a62-95ab-b6fa18c65db9","Type":"ContainerDied","Data":"e8f13b5998bd6d3ce2137a2325229c80dcb3337b6d1480a2fdc3174bb532a65c"} Oct 30 18:41:36.262110 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.262090 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r" event={"ID":"f7d52a97-9fde-4a62-95ab-b6fa18c65db9","Type":"ContainerDied","Data":"d4288c0ddfa79000b580a6461af331da97dd5926c73725d6c95f9fc9b574fbf6"} Oct 30 18:41:36.269350 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.269326 4603 scope.go:122] "RemoveContainer" containerID="32bdcf8d25dd9ebc0df48beb079f3030d10113d3c0ef604e737cdcec61987f6d" Oct 30 18:41:36.269639 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:41:36.269619 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32bdcf8d25dd9ebc0df48beb079f3030d10113d3c0ef604e737cdcec61987f6d\": container with ID starting with 32bdcf8d25dd9ebc0df48beb079f3030d10113d3c0ef604e737cdcec61987f6d not found: ID does not exist" containerID="32bdcf8d25dd9ebc0df48beb079f3030d10113d3c0ef604e737cdcec61987f6d" Oct 30 18:41:36.269735 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.269651 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32bdcf8d25dd9ebc0df48beb079f3030d10113d3c0ef604e737cdcec61987f6d"} err="failed to get container status \"32bdcf8d25dd9ebc0df48beb079f3030d10113d3c0ef604e737cdcec61987f6d\": rpc error: code = NotFound desc = could not find container \"32bdcf8d25dd9ebc0df48beb079f3030d10113d3c0ef604e737cdcec61987f6d\": container with ID starting with 32bdcf8d25dd9ebc0df48beb079f3030d10113d3c0ef604e737cdcec61987f6d not found: ID does not exist" Oct 30 18:41:36.269735 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.269674 4603 scope.go:122] "RemoveContainer" containerID="e8f13b5998bd6d3ce2137a2325229c80dcb3337b6d1480a2fdc3174bb532a65c" Oct 30 18:41:36.279156 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.279132 4603 scope.go:122] "RemoveContainer" containerID="e8f13b5998bd6d3ce2137a2325229c80dcb3337b6d1480a2fdc3174bb532a65c" Oct 30 18:41:36.279484 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:41:36.279462 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8f13b5998bd6d3ce2137a2325229c80dcb3337b6d1480a2fdc3174bb532a65c\": container with ID starting with e8f13b5998bd6d3ce2137a2325229c80dcb3337b6d1480a2fdc3174bb532a65c not found: ID does not exist" containerID="e8f13b5998bd6d3ce2137a2325229c80dcb3337b6d1480a2fdc3174bb532a65c" Oct 30 18:41:36.279615 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.279489 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f13b5998bd6d3ce2137a2325229c80dcb3337b6d1480a2fdc3174bb532a65c"} err="failed to get container status \"e8f13b5998bd6d3ce2137a2325229c80dcb3337b6d1480a2fdc3174bb532a65c\": rpc error: code = NotFound desc = could not find container \"e8f13b5998bd6d3ce2137a2325229c80dcb3337b6d1480a2fdc3174bb532a65c\": container with ID starting with e8f13b5998bd6d3ce2137a2325229c80dcb3337b6d1480a2fdc3174bb532a65c not found: ID does not exist" Oct 30 18:41:36.300258 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.300233 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7d52a97-9fde-4a62-95ab-b6fa18c65db9-kube-api-access-kkd6s\" (UniqueName: \"kubernetes.io/projected/f7d52a97-9fde-4a62-95ab-b6fa18c65db9-kube-api-access-kkd6s\") pod \"f7d52a97-9fde-4a62-95ab-b6fa18c65db9\" (UID: \"f7d52a97-9fde-4a62-95ab-b6fa18c65db9\") " Oct 30 18:41:36.300356 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.300297 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d9f4fc8-48f9-4e12-afa4-fecea8575b5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9f4fc8-48f9-4e12-afa4-fecea8575b5c-plugins-config-volume\") pod \"4d9f4fc8-48f9-4e12-afa4-fecea8575b5c\" (UID: \"4d9f4fc8-48f9-4e12-afa4-fecea8575b5c\") " Oct 30 18:41:36.300433 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.300416 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7d52a97-9fde-4a62-95ab-b6fa18c65db9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d52a97-9fde-4a62-95ab-b6fa18c65db9-plugins-config-volume\") pod \"f7d52a97-9fde-4a62-95ab-b6fa18c65db9\" (UID: \"f7d52a97-9fde-4a62-95ab-b6fa18c65db9\") " Oct 30 18:41:36.300514 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.300481 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d9f4fc8-48f9-4e12-afa4-fecea8575b5c-kube-api-access-f5vqw\" (UniqueName: \"kubernetes.io/projected/4d9f4fc8-48f9-4e12-afa4-fecea8575b5c-kube-api-access-f5vqw\") pod \"4d9f4fc8-48f9-4e12-afa4-fecea8575b5c\" (UID: \"4d9f4fc8-48f9-4e12-afa4-fecea8575b5c\") " Oct 30 18:41:36.300766 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.300739 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d9f4fc8-48f9-4e12-afa4-fecea8575b5c-plugins-config-volume" pod "4d9f4fc8-48f9-4e12-afa4-fecea8575b5c" (UID: "4d9f4fc8-48f9-4e12-afa4-fecea8575b5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:36.300872 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.300773 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d52a97-9fde-4a62-95ab-b6fa18c65db9-plugins-config-volume" pod "f7d52a97-9fde-4a62-95ab-b6fa18c65db9" (UID: "f7d52a97-9fde-4a62-95ab-b6fa18c65db9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:36.300960 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.300942 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9f4fc8-48f9-4e12-afa4-fecea8575b5c-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:36.301019 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.300970 4603 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d52a97-9fde-4a62-95ab-b6fa18c65db9-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:36.302480 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.302458 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d52a97-9fde-4a62-95ab-b6fa18c65db9-kube-api-access-kkd6s" pod "f7d52a97-9fde-4a62-95ab-b6fa18c65db9" (UID: "f7d52a97-9fde-4a62-95ab-b6fa18c65db9"). InnerVolumeSpecName "kube-api-access-kkd6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:36.302622 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.302606 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9f4fc8-48f9-4e12-afa4-fecea8575b5c-kube-api-access-f5vqw" pod "4d9f4fc8-48f9-4e12-afa4-fecea8575b5c" (UID: "4d9f4fc8-48f9-4e12-afa4-fecea8575b5c"). InnerVolumeSpecName "kube-api-access-f5vqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:36.402462 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.402438 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkd6s\" (UniqueName: \"kubernetes.io/projected/f7d52a97-9fde-4a62-95ab-b6fa18c65db9-kube-api-access-kkd6s\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:36.402462 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.402461 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5vqw\" (UniqueName: \"kubernetes.io/projected/4d9f4fc8-48f9-4e12-afa4-fecea8575b5c-kube-api-access-f5vqw\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:36.609699 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.609674 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r"] Oct 30 18:41:36.616553 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.616532 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ac422f-d8899f48d-kmn6r"] Oct 30 18:41:36.649297 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.649267 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw"] Oct 30 18:41:36.654971 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:36.654951 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ac422f-77b7cd85cf-dgqtw"] Oct 30 18:41:37.407989 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:37.407955 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d9f4fc8-48f9-4e12-afa4-fecea8575b5c" path="/var/lib/kubelet/pods/4d9f4fc8-48f9-4e12-afa4-fecea8575b5c/volumes" Oct 30 18:41:37.408386 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:37.408328 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7d52a97-9fde-4a62-95ab-b6fa18c65db9" path="/var/lib/kubelet/pods/f7d52a97-9fde-4a62-95ab-b6fa18c65db9/volumes" Oct 30 18:41:37.834886 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:37.834849 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:38.473793 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:38.473756 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:41:39.449840 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:39.449569 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:40.055528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:40.054937 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl"] Oct 30 18:41:40.177529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:40.177422 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:41.340041 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:41.340011 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:42.949752 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:42.949718 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:44.531590 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:44.531553 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:46.137721 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:46.137679 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:47.756392 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:47.756346 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:49.345852 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:49.345820 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:50.937728 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:50.937693 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:51.929283 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:41:51.929245 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:41:52.645641 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:52.645602 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:54.335524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:54.335470 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:55.078728 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.078690 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" podUID="81d31061-bdab-430c-9dba-d50879b97a29" containerName="main" containerID="cri-o://fd478adb11433e14e16d7e6d98139c62a19cb873c27e2cf3715a9d6e7ed1aead" gracePeriod=45 Oct 30 18:41:55.329945 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.329894 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:41:55.372374 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.372353 4603 generic.go:356] "Generic (PLEG): container finished" podID="81d31061-bdab-430c-9dba-d50879b97a29" containerID="fd478adb11433e14e16d7e6d98139c62a19cb873c27e2cf3715a9d6e7ed1aead" exitCode=0 Oct 30 18:41:55.372787 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.372409 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" Oct 30 18:41:55.372787 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.372411 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" event={"ID":"81d31061-bdab-430c-9dba-d50879b97a29","Type":"ContainerDied","Data":"fd478adb11433e14e16d7e6d98139c62a19cb873c27e2cf3715a9d6e7ed1aead"} Oct 30 18:41:55.372787 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.372559 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl" event={"ID":"81d31061-bdab-430c-9dba-d50879b97a29","Type":"ContainerDied","Data":"451df865a8a92cb9d12ecdca9cefbf2d0774af303bc229a64315b4d4918ca267"} Oct 30 18:41:55.372787 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.372583 4603 scope.go:122] "RemoveContainer" containerID="fd478adb11433e14e16d7e6d98139c62a19cb873c27e2cf3715a9d6e7ed1aead" Oct 30 18:41:55.383390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.383366 4603 scope.go:122] "RemoveContainer" containerID="fd478adb11433e14e16d7e6d98139c62a19cb873c27e2cf3715a9d6e7ed1aead" Oct 30 18:41:55.383716 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:41:55.383692 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd478adb11433e14e16d7e6d98139c62a19cb873c27e2cf3715a9d6e7ed1aead\": container with ID starting with fd478adb11433e14e16d7e6d98139c62a19cb873c27e2cf3715a9d6e7ed1aead not found: ID does not exist" containerID="fd478adb11433e14e16d7e6d98139c62a19cb873c27e2cf3715a9d6e7ed1aead" Oct 30 18:41:55.383814 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.383726 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd478adb11433e14e16d7e6d98139c62a19cb873c27e2cf3715a9d6e7ed1aead"} err="failed to get container status \"fd478adb11433e14e16d7e6d98139c62a19cb873c27e2cf3715a9d6e7ed1aead\": rpc error: code = NotFound desc = could not find container \"fd478adb11433e14e16d7e6d98139c62a19cb873c27e2cf3715a9d6e7ed1aead\": container with ID starting with fd478adb11433e14e16d7e6d98139c62a19cb873c27e2cf3715a9d6e7ed1aead not found: ID does not exist" Oct 30 18:41:55.411099 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.411069 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-home\" (UniqueName: \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-home\") pod \"81d31061-bdab-430c-9dba-d50879b97a29\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " Oct 30 18:41:55.411261 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.411185 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-dshm\") pod \"81d31061-bdab-430c-9dba-d50879b97a29\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " Oct 30 18:41:55.411261 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.411224 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d31061-bdab-430c-9dba-d50879b97a29-kube-api-access-8gqvs\" (UniqueName: \"kubernetes.io/projected/81d31061-bdab-430c-9dba-d50879b97a29-kube-api-access-8gqvs\") pod \"81d31061-bdab-430c-9dba-d50879b97a29\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " Oct 30 18:41:55.411407 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.411301 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81d31061-bdab-430c-9dba-d50879b97a29-tls-certs\" (UniqueName: \"kubernetes.io/secret/81d31061-bdab-430c-9dba-d50879b97a29-tls-certs\") pod \"81d31061-bdab-430c-9dba-d50879b97a29\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " Oct 30 18:41:55.411407 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.411334 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-tmp-dir\") pod \"81d31061-bdab-430c-9dba-d50879b97a29\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " Oct 30 18:41:55.411407 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.411334 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-home" pod "81d31061-bdab-430c-9dba-d50879b97a29" (UID: "81d31061-bdab-430c-9dba-d50879b97a29"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:55.411407 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.411394 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-model-cache\") pod \"81d31061-bdab-430c-9dba-d50879b97a29\" (UID: \"81d31061-bdab-430c-9dba-d50879b97a29\") " Oct 30 18:41:55.411731 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.411699 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-tmp-dir" pod "81d31061-bdab-430c-9dba-d50879b97a29" (UID: "81d31061-bdab-430c-9dba-d50879b97a29"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:55.411832 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.411750 4603 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-home\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:55.412026 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.411968 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-model-cache" pod "81d31061-bdab-430c-9dba-d50879b97a29" (UID: "81d31061-bdab-430c-9dba-d50879b97a29"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:55.414105 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.414070 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81d31061-bdab-430c-9dba-d50879b97a29-tls-certs" pod "81d31061-bdab-430c-9dba-d50879b97a29" (UID: "81d31061-bdab-430c-9dba-d50879b97a29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:55.414265 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.414162 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d31061-bdab-430c-9dba-d50879b97a29-kube-api-access-8gqvs" pod "81d31061-bdab-430c-9dba-d50879b97a29" (UID: "81d31061-bdab-430c-9dba-d50879b97a29"). InnerVolumeSpecName "kube-api-access-8gqvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:55.414427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.414405 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-dshm" pod "81d31061-bdab-430c-9dba-d50879b97a29" (UID: "81d31061-bdab-430c-9dba-d50879b97a29"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:55.512710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.512674 4603 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-tmp-dir\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:55.512710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.512705 4603 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-model-cache\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:55.512710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.512713 4603 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d31061-bdab-430c-9dba-d50879b97a29-dshm\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:55.513017 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.512722 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gqvs\" (UniqueName: \"kubernetes.io/projected/81d31061-bdab-430c-9dba-d50879b97a29-kube-api-access-8gqvs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:55.513017 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.512731 4603 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d31061-bdab-430c-9dba-d50879b97a29-tls-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:41:55.721589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.717611 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl"] Oct 30 18:41:55.722562 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.722538 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-ac422f/praxis-route-ac422f-kserve-85c95f9b9-djkpl"] Oct 30 18:41:55.959903 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:55.959150 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:56.232837 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:56.232755 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:57.408832 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:57.408799 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81d31061-bdab-430c-9dba-d50879b97a29" path="/var/lib/kubelet/pods/81d31061-bdab-430c-9dba-d50879b97a29/volumes" Oct 30 18:41:57.540748 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:57.540700 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:58.054270 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:58.054231 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:59.151069 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:41:59.151028 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:01.065031 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:01.064959 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:02.747425 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:02.747390 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:04.361184 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:04.361138 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:05.953695 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:05.953661 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:07.646170 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:07.646117 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:09.249292 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:09.249256 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:11.025276 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:11.025233 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:12.894383 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:12.894348 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:14.583563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:14.583527 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:16.147955 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:16.147913 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:16.772060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:16.772023 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:17.760205 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:17.760173 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:18.410930 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:18.410897 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:18.897455 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:18.897420 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:19.456557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:19.456523 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:19.618585 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:19.618548 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:20.777395 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:20.777364 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 18:42:21.093885 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:21.093799 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7cgxg_95709b19-3c85-4ef1-9110-c23a97847732/ovn-acl-logging/0.log" Oct 30 18:42:21.209107 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:21.209071 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:22.056926 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:42:22.056871 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:42:22.890920 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:22.890876 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:23.481449 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:23.481420 4603 scope.go:122] "RemoveContainer" containerID="3b374f6038729b8f36db585561f52e7a6e2081fa6224d206960b00ac8934c2af" Oct 30 18:42:24.138879 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:24.138851 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:24.420987 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:24.420903 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:24.504447 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:24.504411 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr"] Oct 30 18:42:24.504981 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:24.504678 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" podUID="ec1892e5-3097-4ce7-8138-caceb31225b8" containerName="istio-proxy" containerID="cri-o://da3fe7c4c0cc06e99c403590fed6447b1ca1780ef1e31d66f0a3ae549525f2f1" gracePeriod=30 Oct 30 18:42:24.614780 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:24.614738 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:26.348444 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:26.348409 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:28.043955 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:28.043917 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:29.824910 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.824885 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:42:29.888969 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.888941 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec1892e5-3097-4ce7-8138-caceb31225b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1892e5-3097-4ce7-8138-caceb31225b8-istiod-ca-cert\") pod \"ec1892e5-3097-4ce7-8138-caceb31225b8\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " Oct 30 18:42:29.889136 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.888989 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-workload-socket\") pod \"ec1892e5-3097-4ce7-8138-caceb31225b8\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " Oct 30 18:42:29.889136 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.889033 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-envoy\") pod \"ec1892e5-3097-4ce7-8138-caceb31225b8\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " Oct 30 18:42:29.889136 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.889069 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec1892e5-3097-4ce7-8138-caceb31225b8-kube-api-access-f62m6\" (UniqueName: \"kubernetes.io/projected/ec1892e5-3097-4ce7-8138-caceb31225b8-kube-api-access-f62m6\") pod \"ec1892e5-3097-4ce7-8138-caceb31225b8\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " Oct 30 18:42:29.889323 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.889152 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-workload-certs\") pod \"ec1892e5-3097-4ce7-8138-caceb31225b8\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " Oct 30 18:42:29.889323 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.889211 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-data\") pod \"ec1892e5-3097-4ce7-8138-caceb31225b8\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " Oct 30 18:42:29.889323 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.889241 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-token\" (UniqueName: \"kubernetes.io/projected/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-token\") pod \"ec1892e5-3097-4ce7-8138-caceb31225b8\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " Oct 30 18:42:29.889323 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.889278 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-podinfo\") pod \"ec1892e5-3097-4ce7-8138-caceb31225b8\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " Oct 30 18:42:29.889323 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.889297 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-workload-socket" pod "ec1892e5-3097-4ce7-8138-caceb31225b8" (UID: "ec1892e5-3097-4ce7-8138-caceb31225b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:29.889612 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.889325 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-credential-socket\") pod \"ec1892e5-3097-4ce7-8138-caceb31225b8\" (UID: \"ec1892e5-3097-4ce7-8138-caceb31225b8\") " Oct 30 18:42:29.889612 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.889390 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1892e5-3097-4ce7-8138-caceb31225b8-istiod-ca-cert" pod "ec1892e5-3097-4ce7-8138-caceb31225b8" (UID: "ec1892e5-3097-4ce7-8138-caceb31225b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:29.889715 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.889658 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-data" pod "ec1892e5-3097-4ce7-8138-caceb31225b8" (UID: "ec1892e5-3097-4ce7-8138-caceb31225b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:29.889781 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.889769 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1892e5-3097-4ce7-8138-caceb31225b8-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.889847 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.889790 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.889847 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.889806 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.889971 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.889892 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-credential-socket" pod "ec1892e5-3097-4ce7-8138-caceb31225b8" (UID: "ec1892e5-3097-4ce7-8138-caceb31225b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:29.889971 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.889952 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-workload-certs" pod "ec1892e5-3097-4ce7-8138-caceb31225b8" (UID: "ec1892e5-3097-4ce7-8138-caceb31225b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:29.892199 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.892174 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-podinfo" pod "ec1892e5-3097-4ce7-8138-caceb31225b8" (UID: "ec1892e5-3097-4ce7-8138-caceb31225b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:42:29.892333 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.892242 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-envoy" pod "ec1892e5-3097-4ce7-8138-caceb31225b8" (UID: "ec1892e5-3097-4ce7-8138-caceb31225b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:29.892532 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.892483 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1892e5-3097-4ce7-8138-caceb31225b8-kube-api-access-f62m6" pod "ec1892e5-3097-4ce7-8138-caceb31225b8" (UID: "ec1892e5-3097-4ce7-8138-caceb31225b8"). InnerVolumeSpecName "kube-api-access-f62m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:29.892644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.892566 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-token" pod "ec1892e5-3097-4ce7-8138-caceb31225b8" (UID: "ec1892e5-3097-4ce7-8138-caceb31225b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:29.979632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.979605 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:29.990690 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.990670 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.990690 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.990689 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f62m6\" (UniqueName: \"kubernetes.io/projected/ec1892e5-3097-4ce7-8138-caceb31225b8-kube-api-access-f62m6\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.990818 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.990699 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.990818 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.990710 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.990818 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.990717 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1892e5-3097-4ce7-8138-caceb31225b8-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.990818 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:29.990725 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1892e5-3097-4ce7-8138-caceb31225b8-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:42:30.670870 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:30.670840 4603 generic.go:356] "Generic (PLEG): container finished" podID="ec1892e5-3097-4ce7-8138-caceb31225b8" containerID="da3fe7c4c0cc06e99c403590fed6447b1ca1780ef1e31d66f0a3ae549525f2f1" exitCode=0 Oct 30 18:42:30.671054 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:30.670889 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" Oct 30 18:42:30.671054 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:30.670931 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" event={"ID":"ec1892e5-3097-4ce7-8138-caceb31225b8","Type":"ContainerDied","Data":"da3fe7c4c0cc06e99c403590fed6447b1ca1780ef1e31d66f0a3ae549525f2f1"} Oct 30 18:42:30.671054 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:30.670986 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr" event={"ID":"ec1892e5-3097-4ce7-8138-caceb31225b8","Type":"ContainerDied","Data":"f29a41298a361cab25093cd80c1754972a5bef66e7d5e4e06b03cfda3636ac63"} Oct 30 18:42:30.671054 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:30.671009 4603 scope.go:122] "RemoveContainer" containerID="da3fe7c4c0cc06e99c403590fed6447b1ca1780ef1e31d66f0a3ae549525f2f1" Oct 30 18:42:30.680851 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:30.680835 4603 scope.go:122] "RemoveContainer" containerID="90c73c94b47ccb54723aa8dbb9d8b36ad0c10d75451978cd5531774413ab2854" Oct 30 18:42:30.690907 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:30.690888 4603 scope.go:122] "RemoveContainer" containerID="da3fe7c4c0cc06e99c403590fed6447b1ca1780ef1e31d66f0a3ae549525f2f1" Oct 30 18:42:30.691165 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:42:30.691147 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da3fe7c4c0cc06e99c403590fed6447b1ca1780ef1e31d66f0a3ae549525f2f1\": container with ID starting with da3fe7c4c0cc06e99c403590fed6447b1ca1780ef1e31d66f0a3ae549525f2f1 not found: ID does not exist" containerID="da3fe7c4c0cc06e99c403590fed6447b1ca1780ef1e31d66f0a3ae549525f2f1" Oct 30 18:42:30.691213 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:30.691174 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da3fe7c4c0cc06e99c403590fed6447b1ca1780ef1e31d66f0a3ae549525f2f1"} err="failed to get container status \"da3fe7c4c0cc06e99c403590fed6447b1ca1780ef1e31d66f0a3ae549525f2f1\": rpc error: code = NotFound desc = could not find container \"da3fe7c4c0cc06e99c403590fed6447b1ca1780ef1e31d66f0a3ae549525f2f1\": container with ID starting with da3fe7c4c0cc06e99c403590fed6447b1ca1780ef1e31d66f0a3ae549525f2f1 not found: ID does not exist" Oct 30 18:42:30.691213 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:30.691190 4603 scope.go:122] "RemoveContainer" containerID="90c73c94b47ccb54723aa8dbb9d8b36ad0c10d75451978cd5531774413ab2854" Oct 30 18:42:30.691448 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:42:30.691433 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c73c94b47ccb54723aa8dbb9d8b36ad0c10d75451978cd5531774413ab2854\": container with ID starting with 90c73c94b47ccb54723aa8dbb9d8b36ad0c10d75451978cd5531774413ab2854 not found: ID does not exist" containerID="90c73c94b47ccb54723aa8dbb9d8b36ad0c10d75451978cd5531774413ab2854" Oct 30 18:42:30.691516 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:30.691452 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c73c94b47ccb54723aa8dbb9d8b36ad0c10d75451978cd5531774413ab2854"} err="failed to get container status \"90c73c94b47ccb54723aa8dbb9d8b36ad0c10d75451978cd5531774413ab2854\": rpc error: code = NotFound desc = could not find container \"90c73c94b47ccb54723aa8dbb9d8b36ad0c10d75451978cd5531774413ab2854\": container with ID starting with 90c73c94b47ccb54723aa8dbb9d8b36ad0c10d75451978cd5531774413ab2854 not found: ID does not exist" Oct 30 18:42:30.733517 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:30.733474 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr"] Oct 30 18:42:30.737738 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:30.737717 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-npnpr"] Oct 30 18:42:31.407936 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:31.407897 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec1892e5-3097-4ce7-8138-caceb31225b8" path="/var/lib/kubelet/pods/ec1892e5-3097-4ce7-8138-caceb31225b8/volumes" Oct 30 18:42:31.663795 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:31.663714 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:33.373303 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:33.373090 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:35.045719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:35.045687 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:35.980535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:35.969846 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:36.123822 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:36.123776 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:36.903030 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:36.902998 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:38.548563 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:38.548525 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:39.972048 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:39.972008 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:40.437102 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:40.437063 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:40.573771 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:40.573738 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:41.199445 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:41.199411 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:42.361433 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:42.361397 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:44.276125 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:44.276090 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:45.954338 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:45.954305 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:47.583939 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:47.583903 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:47.774402 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:47.774364 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:47.914192 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:47.914103 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:48.147260 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:48.147225 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv"] Oct 30 18:42:48.147536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:48.147492 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" podUID="b6ce7227-01e3-486c-93eb-f945903f32cf" containerName="istio-proxy" containerID="cri-o://a8718679c05bb9ac84c92c970d87ec8e374f27644d90aa3abe19df52bd623562" gracePeriod=30 Oct 30 18:42:48.212898 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:48.212830 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:49.262340 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.262301 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:49.411747 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.411712 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:49.790899 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.790868 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5"] Oct 30 18:42:49.791236 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791224 4603 cpu_manager.go:414] "containerMap: removing container" podUID="5411300d-c1aa-4397-be3d-0ce869a1c1cb" containerName="istio-proxy" Oct 30 18:42:49.791294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791237 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5411300d-c1aa-4397-be3d-0ce869a1c1cb" containerName="istio-proxy" Oct 30 18:42:49.791294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791248 4603 cpu_manager.go:414] "containerMap: removing container" podUID="b664f4ca-736e-4cb1-991d-985d1e6aff34" containerName="istio-proxy" Oct 30 18:42:49.791294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791253 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b664f4ca-736e-4cb1-991d-985d1e6aff34" containerName="istio-proxy" Oct 30 18:42:49.791294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791264 4603 cpu_manager.go:414] "containerMap: removing container" podUID="187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" containerName="istio-proxy" Oct 30 18:42:49.791294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791269 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" containerName="istio-proxy" Oct 30 18:42:49.791294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791276 4603 cpu_manager.go:414] "containerMap: removing container" podUID="4d9f4fc8-48f9-4e12-afa4-fecea8575b5c" containerName="payload-pre-processing" Oct 30 18:42:49.791294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791280 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d9f4fc8-48f9-4e12-afa4-fecea8575b5c" containerName="payload-pre-processing" Oct 30 18:42:49.791294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791287 4603 cpu_manager.go:414] "containerMap: removing container" podUID="d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" containerName="istio-proxy" Oct 30 18:42:49.791294 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791292 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791303 4603 cpu_manager.go:414] "containerMap: removing container" podUID="22e0b280-2b65-4fad-8344-67049190d108" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791307 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22e0b280-2b65-4fad-8344-67049190d108" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791313 4603 cpu_manager.go:414] "containerMap: removing container" podUID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791319 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791325 4603 cpu_manager.go:414] "containerMap: removing container" podUID="db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791331 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791339 4603 cpu_manager.go:414] "containerMap: removing container" podUID="9e2e8d0b-1be6-4e07-aa98-db2472756ee5" containerName="payload-pre-processing" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791343 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e2e8d0b-1be6-4e07-aa98-db2472756ee5" containerName="payload-pre-processing" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791350 4603 cpu_manager.go:414] "containerMap: removing container" podUID="81d31061-bdab-430c-9dba-d50879b97a29" containerName="main" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791354 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d31061-bdab-430c-9dba-d50879b97a29" containerName="main" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791361 4603 cpu_manager.go:414] "containerMap: removing container" podUID="d4a3adf6-fb23-4334-8bab-f800b4e8e089" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791365 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4a3adf6-fb23-4334-8bab-f800b4e8e089" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791371 4603 cpu_manager.go:414] "containerMap: removing container" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791376 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791385 4603 cpu_manager.go:414] "containerMap: removing container" podUID="e5c8b1f8-fba8-4afc-8887-8b1375d67b22" containerName="maas-api" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791390 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c8b1f8-fba8-4afc-8887-8b1375d67b22" containerName="maas-api" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791396 4603 cpu_manager.go:414] "containerMap: removing container" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791401 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791408 4603 cpu_manager.go:414] "containerMap: removing container" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791411 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791420 4603 cpu_manager.go:414] "containerMap: removing container" podUID="0815ac39-de71-4bc3-98d3-28c88a9d2139" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791424 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0815ac39-de71-4bc3-98d3-28c88a9d2139" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791432 4603 cpu_manager.go:414] "containerMap: removing container" podUID="ec3c95aa-215c-4f15-8dc8-bc0beb954236" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791436 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec3c95aa-215c-4f15-8dc8-bc0beb954236" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791442 4603 cpu_manager.go:414] "containerMap: removing container" podUID="73366d41-4910-43f0-aad8-2542401ec0f6" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791446 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73366d41-4910-43f0-aad8-2542401ec0f6" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791455 4603 cpu_manager.go:414] "containerMap: removing container" podUID="c41e57c9-3c80-40fc-8ab4-8b3df57b0136" containerName="payload-processing" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791459 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c41e57c9-3c80-40fc-8ab4-8b3df57b0136" containerName="payload-processing" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791465 4603 cpu_manager.go:414] "containerMap: removing container" podUID="58e2469b-dc0d-40ab-bd74-a47c4a9e438b" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791469 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e2469b-dc0d-40ab-bd74-a47c4a9e438b" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791474 4603 cpu_manager.go:414] "containerMap: removing container" podUID="dba942a9-7d94-415a-8491-a17212be1601" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791480 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dba942a9-7d94-415a-8491-a17212be1601" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791486 4603 cpu_manager.go:414] "containerMap: removing container" podUID="dba942a9-7d94-415a-8491-a17212be1601" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791491 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dba942a9-7d94-415a-8491-a17212be1601" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791526 4603 cpu_manager.go:414] "containerMap: removing container" podUID="d2888d82-f0fe-49af-a23d-c97b4f30be70" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791534 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2888d82-f0fe-49af-a23d-c97b4f30be70" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791543 4603 cpu_manager.go:414] "containerMap: removing container" podUID="f9bcc5bf-0143-429f-95c1-298df70d5f88" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791550 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9bcc5bf-0143-429f-95c1-298df70d5f88" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791559 4603 cpu_manager.go:414] "containerMap: removing container" podUID="d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791565 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791575 4603 cpu_manager.go:414] "containerMap: removing container" podUID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791581 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791592 4603 cpu_manager.go:414] "containerMap: removing container" podUID="d5567046-aa24-421a-8339-fdd9cf56e62c" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791600 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5567046-aa24-421a-8339-fdd9cf56e62c" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791610 4603 cpu_manager.go:414] "containerMap: removing container" podUID="a5a9b344-c8bb-433b-a6a2-e62c357f4101" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791616 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a9b344-c8bb-433b-a6a2-e62c357f4101" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791629 4603 cpu_manager.go:414] "containerMap: removing container" podUID="22e0b280-2b65-4fad-8344-67049190d108" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791636 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22e0b280-2b65-4fad-8344-67049190d108" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791647 4603 cpu_manager.go:414] "containerMap: removing container" podUID="b6e43478-8cc1-4452-8bc8-fed14f127896" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791655 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6e43478-8cc1-4452-8bc8-fed14f127896" containerName="istio-proxy" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791663 4603 cpu_manager.go:414] "containerMap: removing container" podUID="f7d52a97-9fde-4a62-95ab-b6fa18c65db9" containerName="payload-processing" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791670 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d52a97-9fde-4a62-95ab-b6fa18c65db9" containerName="payload-processing" Oct 30 18:42:49.791655 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791683 4603 cpu_manager.go:414] "containerMap: removing container" podUID="73366d41-4910-43f0-aad8-2542401ec0f6" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791688 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73366d41-4910-43f0-aad8-2542401ec0f6" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791697 4603 cpu_manager.go:414] "containerMap: removing container" podUID="70342038-ca6c-4a23-980b-7b750af92367" containerName="payload-processing" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791702 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70342038-ca6c-4a23-980b-7b750af92367" containerName="payload-processing" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791708 4603 cpu_manager.go:414] "containerMap: removing container" podUID="d79321de-091c-47e7-8320-3471ef8f554b" containerName="maas-api" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791712 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d79321de-091c-47e7-8320-3471ef8f554b" containerName="maas-api" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791718 4603 cpu_manager.go:414] "containerMap: removing container" podUID="1d187b51-f5c6-4a63-8300-2f01184a15d3" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791723 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d187b51-f5c6-4a63-8300-2f01184a15d3" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791728 4603 cpu_manager.go:414] "containerMap: removing container" podUID="0baa3ccf-73e5-4555-9e51-5c627b74124d" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791732 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0baa3ccf-73e5-4555-9e51-5c627b74124d" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791739 4603 cpu_manager.go:414] "containerMap: removing container" podUID="cb49f732-be41-4ed9-b7e6-a4eb39e146e2" containerName="maas-api" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791743 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb49f732-be41-4ed9-b7e6-a4eb39e146e2" containerName="maas-api" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791755 4603 cpu_manager.go:414] "containerMap: removing container" podUID="0a8b0573-4657-4fc6-b7fd-43d8e882062d" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791763 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a8b0573-4657-4fc6-b7fd-43d8e882062d" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791771 4603 cpu_manager.go:414] "containerMap: removing container" podUID="e0c05c4b-da86-4622-88ec-960248babce5" containerName="maas-api" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791778 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c05c4b-da86-4622-88ec-960248babce5" containerName="maas-api" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791786 4603 cpu_manager.go:414] "containerMap: removing container" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791793 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791799 4603 cpu_manager.go:414] "containerMap: removing container" podUID="c7be351b-62cc-4555-84fa-a0f018a99059" containerName="payload-processing" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791803 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7be351b-62cc-4555-84fa-a0f018a99059" containerName="payload-processing" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791811 4603 cpu_manager.go:414] "containerMap: removing container" podUID="f308b20b-d5d6-41db-8376-9cb9413b13cf" containerName="payload-pre-processing" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791817 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f308b20b-d5d6-41db-8376-9cb9413b13cf" containerName="payload-pre-processing" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791827 4603 cpu_manager.go:414] "containerMap: removing container" podUID="d99bf6c9-a08f-4f49-b845-0f411f1f1720" containerName="payload-pre-processing" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791835 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d99bf6c9-a08f-4f49-b845-0f411f1f1720" containerName="payload-pre-processing" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791846 4603 cpu_manager.go:414] "containerMap: removing container" podUID="d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791853 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791863 4603 cpu_manager.go:414] "containerMap: removing container" podUID="47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791871 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791881 4603 cpu_manager.go:414] "containerMap: removing container" podUID="ec1892e5-3097-4ce7-8138-caceb31225b8" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791886 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1892e5-3097-4ce7-8138-caceb31225b8" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791893 4603 cpu_manager.go:414] "containerMap: removing container" podUID="0baa3ccf-73e5-4555-9e51-5c627b74124d" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791897 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0baa3ccf-73e5-4555-9e51-5c627b74124d" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791905 4603 cpu_manager.go:414] "containerMap: removing container" podUID="f7882513-ef6d-4b11-871e-39a29ad022df" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791910 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7882513-ef6d-4b11-871e-39a29ad022df" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791916 4603 cpu_manager.go:414] "containerMap: removing container" podUID="c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791920 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791925 4603 cpu_manager.go:414] "containerMap: removing container" podUID="ec1892e5-3097-4ce7-8138-caceb31225b8" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.791930 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1892e5-3097-4ce7-8138-caceb31225b8" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792005 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec3c95aa-215c-4f15-8dc8-bc0beb954236" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792012 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="73366d41-4910-43f0-aad8-2542401ec0f6" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792018 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="c41e57c9-3c80-40fc-8ab4-8b3df57b0136" containerName="payload-processing" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792024 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="5411300d-c1aa-4397-be3d-0ce869a1c1cb" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792030 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e2469b-dc0d-40ab-bd74-a47c4a9e438b" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792035 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="dba942a9-7d94-415a-8491-a17212be1601" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792043 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="dba942a9-7d94-415a-8491-a17212be1601" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792048 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2888d82-f0fe-49af-a23d-c97b4f30be70" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792054 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9bcc5bf-0143-429f-95c1-298df70d5f88" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792059 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792064 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792070 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5567046-aa24-421a-8339-fdd9cf56e62c" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792076 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a9b344-c8bb-433b-a6a2-e62c357f4101" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792083 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="22e0b280-2b65-4fad-8344-67049190d108" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792092 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6e43478-8cc1-4452-8bc8-fed14f127896" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792098 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d52a97-9fde-4a62-95ab-b6fa18c65db9" containerName="payload-processing" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792103 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="0815ac39-de71-4bc3-98d3-28c88a9d2139" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792110 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="70342038-ca6c-4a23-980b-7b750af92367" containerName="payload-processing" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792115 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="d79321de-091c-47e7-8320-3471ef8f554b" containerName="maas-api" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792122 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d187b51-f5c6-4a63-8300-2f01184a15d3" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792126 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="0baa3ccf-73e5-4555-9e51-5c627b74124d" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792132 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb49f732-be41-4ed9-b7e6-a4eb39e146e2" containerName="maas-api" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792138 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a8b0573-4657-4fc6-b7fd-43d8e882062d" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792143 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c05c4b-da86-4622-88ec-960248babce5" containerName="maas-api" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792148 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcebcb0f-0443-4e43-9850-e4bb46d279f4" containerName="main" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792152 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7be351b-62cc-4555-84fa-a0f018a99059" containerName="payload-processing" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792162 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="f308b20b-d5d6-41db-8376-9cb9413b13cf" containerName="payload-pre-processing" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792171 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="d99bf6c9-a08f-4f49-b845-0f411f1f1720" containerName="payload-pre-processing" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792178 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b7d3ed-4e42-4eb6-abfd-ae968273d8b5" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792184 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="47cefcfc-ddae-4c02-ad2c-ae2cb8769e64" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792189 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1892e5-3097-4ce7-8138-caceb31225b8" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792194 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="0baa3ccf-73e5-4555-9e51-5c627b74124d" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792199 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7882513-ef6d-4b11-871e-39a29ad022df" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792204 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e7acbe-39d1-478a-aaf6-ad5f2b1091f8" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792209 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1892e5-3097-4ce7-8138-caceb31225b8" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792214 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="b664f4ca-736e-4cb1-991d-985d1e6aff34" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792219 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="187c6c98-bcb4-4e66-b394-f5ce4e4c5d9e" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792223 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d9f4fc8-48f9-4e12-afa4-fecea8575b5c" containerName="payload-pre-processing" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792228 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5314716-efc8-4ce5-843f-fb4f5f2ba7e3" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792235 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="22e0b280-2b65-4fad-8344-67049190d108" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792242 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="db8b0a7a-6aef-4237-acb3-c9e2ee5112fd" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792251 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e2e8d0b-1be6-4e07-aa98-db2472756ee5" containerName="payload-pre-processing" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792256 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d31061-bdab-430c-9dba-d50879b97a29" containerName="main" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792261 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4a3adf6-fb23-4334-8bab-f800b4e8e089" containerName="istio-proxy" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792266 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="521cc860-6923-45a7-8823-4859624c9267" containerName="main" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792272 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c8b1f8-fba8-4afc-8887-8b1375d67b22" containerName="maas-api" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792277 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0d5196-98c7-4b7c-b045-623b42c2e3d0" containerName="main" Oct 30 18:42:49.793390 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.792283 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d8dda47-f665-4fcb-bdae-3430e419841c" containerName="main" Oct 30 18:42:49.797433 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.797414 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:49.828324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.828296 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-366d9f-openshift-default-dockercfg-nbrj6\"" Oct 30 18:42:49.831462 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.831434 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5"] Oct 30 18:42:49.878940 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.878807 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:49.929411 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.929375 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-workload-certs\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:49.929640 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.929423 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-workload-socket\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:49.929640 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.929537 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-token\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:49.929640 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.929622 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-credential-socket\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:49.929820 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.929654 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg8jx\" (UniqueName: \"kubernetes.io/projected/0d5bcd9f-9b13-40b9-9922-dadc505730d3-kube-api-access-dg8jx\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:49.929820 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.929672 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-data\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:49.929820 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.929697 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-podinfo\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:49.929820 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.929724 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istiod-ca-cert\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:49.929820 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:49.929750 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-envoy\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:50.030864 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.030827 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-workload-certs\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:50.031093 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.030886 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-workload-socket\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:50.031093 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.030928 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-token\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:50.031093 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.030968 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-credential-socket\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:50.031093 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.031020 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg8jx\" (UniqueName: \"kubernetes.io/projected/0d5bcd9f-9b13-40b9-9922-dadc505730d3-kube-api-access-dg8jx\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:50.031093 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.031048 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-data\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:50.031093 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.031074 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-podinfo\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:50.031430 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.031113 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istiod-ca-cert\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:50.031430 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.031157 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-envoy\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:50.031430 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.031321 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-workload-certs\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:50.031602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.031486 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-workload-socket\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:50.031602 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.031542 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-credential-socket\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:50.031740 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.031720 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-data\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:50.032008 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.031985 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istiod-ca-cert\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:50.033730 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.033708 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-envoy\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:50.033895 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.033784 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-podinfo\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:50.033972 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.033934 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:50.053726 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.053672 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-token\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:50.054451 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.054433 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg8jx\" (UniqueName: \"kubernetes.io/projected/0d5bcd9f-9b13-40b9-9922-dadc505730d3-kube-api-access-dg8jx\") pod \"e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:50.110185 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.110146 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:50.440119 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.440084 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5"] Oct 30 18:42:50.442907 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:42:50.442867 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5bcd9f_9b13_40b9_9922_dadc505730d3.slice/crio-f5d116cd5310200b619805f95ad1150c031cd10e6519b4fe95fd08eed6dd9e38 WatchSource:0}: Error finding container f5d116cd5310200b619805f95ad1150c031cd10e6519b4fe95fd08eed6dd9e38: Status 404 returned error can't find the container with id f5d116cd5310200b619805f95ad1150c031cd10e6519b4fe95fd08eed6dd9e38 Oct 30 18:42:50.445981 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.445940 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:42:50.446102 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.446034 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:42:50.446102 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.446085 4603 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:42:50.806637 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.806600 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" event={"ID":"0d5bcd9f-9b13-40b9-9922-dadc505730d3","Type":"ContainerStarted","Data":"3efe6cc562c99701dc3967cbe3cbc53fb0a306cd6c21e7e294c9d5b8199f5f95"} Oct 30 18:42:50.807642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.807610 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" event={"ID":"0d5bcd9f-9b13-40b9-9922-dadc505730d3","Type":"ContainerStarted","Data":"f5d116cd5310200b619805f95ad1150c031cd10e6519b4fe95fd08eed6dd9e38"} Oct 30 18:42:50.856877 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:50.856804 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" podStartSLOduration=1.856783992 podStartE2EDuration="1.856783992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:50.853419422 +0000 UTC m=+3031.568989308" watchObservedRunningTime="2026-10-30 18:42:50.856783992 +0000 UTC m=+3031.572353869" Oct 30 18:42:51.042206 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:51.042172 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:51.111923 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:51.111837 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:51.147819 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:51.147787 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:51.194031 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:51.193998 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:42:51.354060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:51.354026 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:51.511597 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:51.511458 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:51.692186 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:51.692137 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:51.819332 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:51.819300 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:51.828758 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:51.828731 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:42:52.119688 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:42:52.119187 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:42:52.850748 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:52.850714 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn"] Oct 30 18:42:52.851311 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:52.851284 4603 cpu_manager.go:414] "containerMap: removing container" podUID="5411300d-c1aa-4397-be3d-0ce869a1c1cb" containerName="istio-proxy" Oct 30 18:42:52.851311 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:52.851308 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5411300d-c1aa-4397-be3d-0ce869a1c1cb" containerName="istio-proxy" Oct 30 18:42:52.851434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:52.851340 4603 cpu_manager.go:414] "containerMap: removing container" podUID="0815ac39-de71-4bc3-98d3-28c88a9d2139" containerName="istio-proxy" Oct 30 18:42:52.851434 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:52.851349 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0815ac39-de71-4bc3-98d3-28c88a9d2139" containerName="istio-proxy" Oct 30 18:42:52.851535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:52.851457 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="73366d41-4910-43f0-aad8-2542401ec0f6" containerName="istio-proxy" Oct 30 18:42:52.851535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:52.851470 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="5411300d-c1aa-4397-be3d-0ce869a1c1cb" containerName="istio-proxy" Oct 30 18:42:52.851535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:52.851480 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="744e9a72-c23c-4b8f-b0fa-ef7f791ce58a" containerName="istio-proxy" Oct 30 18:42:52.851535 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:52.851493 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="0815ac39-de71-4bc3-98d3-28c88a9d2139" containerName="istio-proxy" Oct 30 18:42:52.854817 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:52.854800 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:42:52.880708 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:52.880681 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:52.884238 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:52.884215 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-366d9f\"/\"noten-model-366d9f-kserve-self-signed-certs\"" Oct 30 18:42:52.884385 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:52.884286 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-366d9f\"/\"kube-root-ca.crt\"" Oct 30 18:42:52.884438 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:52.884287 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-366d9f\"/\"openshift-service-ca.crt\"" Oct 30 18:42:52.884438 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:52.884303 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-366d9f\"/\"default-dockercfg-58cbj\"" Oct 30 18:42:52.887604 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:52.887583 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn"] Oct 30 18:42:52.919632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:52.919609 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-tls-certs\") pod \"noten-model-366d9f-kserve-76484d4f8f-l7srn\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:42:52.919760 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:52.919737 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-model-cache\") pod \"noten-model-366d9f-kserve-76484d4f8f-l7srn\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:42:52.919835 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:52.919779 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-dshm\") pod \"noten-model-366d9f-kserve-76484d4f8f-l7srn\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:42:52.919835 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:52.919796 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-tmp-dir\") pod \"noten-model-366d9f-kserve-76484d4f8f-l7srn\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:42:52.919835 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:52.919817 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-home\") pod \"noten-model-366d9f-kserve-76484d4f8f-l7srn\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:42:52.920023 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:52.919841 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6qhq\" (UniqueName: \"kubernetes.io/projected/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-kube-api-access-l6qhq\") pod \"noten-model-366d9f-kserve-76484d4f8f-l7srn\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:42:53.020419 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.020392 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-tls-certs\") pod \"noten-model-366d9f-kserve-76484d4f8f-l7srn\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:42:53.020559 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.020471 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-model-cache\") pod \"noten-model-366d9f-kserve-76484d4f8f-l7srn\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:42:53.020610 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.020563 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-dshm\") pod \"noten-model-366d9f-kserve-76484d4f8f-l7srn\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:42:53.020610 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.020589 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-tmp-dir\") pod \"noten-model-366d9f-kserve-76484d4f8f-l7srn\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:42:53.020720 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.020620 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-home\") pod \"noten-model-366d9f-kserve-76484d4f8f-l7srn\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:42:53.020720 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.020647 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6qhq\" (UniqueName: \"kubernetes.io/projected/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-kube-api-access-l6qhq\") pod \"noten-model-366d9f-kserve-76484d4f8f-l7srn\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:42:53.020895 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.020873 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-model-cache\") pod \"noten-model-366d9f-kserve-76484d4f8f-l7srn\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:42:53.020971 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.020955 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-tmp-dir\") pod \"noten-model-366d9f-kserve-76484d4f8f-l7srn\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:42:53.021073 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.021054 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-home\") pod \"noten-model-366d9f-kserve-76484d4f8f-l7srn\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:42:53.023062 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.023036 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-dshm\") pod \"noten-model-366d9f-kserve-76484d4f8f-l7srn\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:42:53.023459 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.023440 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-tls-certs\") pod \"noten-model-366d9f-kserve-76484d4f8f-l7srn\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:42:53.037128 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.037101 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:53.054115 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.054091 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6qhq\" (UniqueName: \"kubernetes.io/projected/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-kube-api-access-l6qhq\") pod \"noten-model-366d9f-kserve-76484d4f8f-l7srn\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:42:53.168791 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.168762 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:42:53.392119 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.392093 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:42:53.479872 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.479851 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn"] Oct 30 18:42:53.481994 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:42:53.481967 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bda071c_1825_4d2e_ab67_2a4acc90a8eb.slice/crio-4937eff1fba3ccab0ff03884fe836e6d276888f5129442cd62f8da3228932eab WatchSource:0}: Error finding container 4937eff1fba3ccab0ff03884fe836e6d276888f5129442cd62f8da3228932eab: Status 404 returned error can't find the container with id 4937eff1fba3ccab0ff03884fe836e6d276888f5129442cd62f8da3228932eab Oct 30 18:42:53.526472 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.526449 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-workload-certs\") pod \"b6ce7227-01e3-486c-93eb-f945903f32cf\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " Oct 30 18:42:53.526589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.526511 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-token\" (UniqueName: \"kubernetes.io/projected/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-token\") pod \"b6ce7227-01e3-486c-93eb-f945903f32cf\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " Oct 30 18:42:53.526589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.526532 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6ce7227-01e3-486c-93eb-f945903f32cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6ce7227-01e3-486c-93eb-f945903f32cf-istiod-ca-cert\") pod \"b6ce7227-01e3-486c-93eb-f945903f32cf\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " Oct 30 18:42:53.526669 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.526644 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-credential-socket\") pod \"b6ce7227-01e3-486c-93eb-f945903f32cf\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " Oct 30 18:42:53.526709 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.526689 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-envoy\") pod \"b6ce7227-01e3-486c-93eb-f945903f32cf\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " Oct 30 18:42:53.526750 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.526725 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-data\") pod \"b6ce7227-01e3-486c-93eb-f945903f32cf\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " Oct 30 18:42:53.526796 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.526752 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-workload-socket\") pod \"b6ce7227-01e3-486c-93eb-f945903f32cf\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " Oct 30 18:42:53.526860 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.526809 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6ce7227-01e3-486c-93eb-f945903f32cf-kube-api-access-vqvf7\" (UniqueName: \"kubernetes.io/projected/b6ce7227-01e3-486c-93eb-f945903f32cf-kube-api-access-vqvf7\") pod \"b6ce7227-01e3-486c-93eb-f945903f32cf\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " Oct 30 18:42:53.526860 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.526847 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-podinfo\") pod \"b6ce7227-01e3-486c-93eb-f945903f32cf\" (UID: \"b6ce7227-01e3-486c-93eb-f945903f32cf\") " Oct 30 18:42:53.526971 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.526854 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-workload-certs" pod "b6ce7227-01e3-486c-93eb-f945903f32cf" (UID: "b6ce7227-01e3-486c-93eb-f945903f32cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:53.526971 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.526905 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-credential-socket" pod "b6ce7227-01e3-486c-93eb-f945903f32cf" (UID: "b6ce7227-01e3-486c-93eb-f945903f32cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:53.526971 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.526917 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ce7227-01e3-486c-93eb-f945903f32cf-istiod-ca-cert" pod "b6ce7227-01e3-486c-93eb-f945903f32cf" (UID: "b6ce7227-01e3-486c-93eb-f945903f32cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:53.527119 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.527079 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-workload-socket" pod "b6ce7227-01e3-486c-93eb-f945903f32cf" (UID: "b6ce7227-01e3-486c-93eb-f945903f32cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:53.527260 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.527187 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-data" pod "b6ce7227-01e3-486c-93eb-f945903f32cf" (UID: "b6ce7227-01e3-486c-93eb-f945903f32cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:53.527260 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.527297 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6ce7227-01e3-486c-93eb-f945903f32cf-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:42:53.527260 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.527315 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:42:53.527260 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.527328 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:42:53.527643 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.527341 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:42:53.527643 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.527353 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:42:53.528799 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.528778 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-token" pod "b6ce7227-01e3-486c-93eb-f945903f32cf" (UID: "b6ce7227-01e3-486c-93eb-f945903f32cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:53.529104 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.529089 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ce7227-01e3-486c-93eb-f945903f32cf-kube-api-access-vqvf7" pod "b6ce7227-01e3-486c-93eb-f945903f32cf" (UID: "b6ce7227-01e3-486c-93eb-f945903f32cf"). InnerVolumeSpecName "kube-api-access-vqvf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:53.529650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.529629 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-envoy" pod "b6ce7227-01e3-486c-93eb-f945903f32cf" (UID: "b6ce7227-01e3-486c-93eb-f945903f32cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:53.529650 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.529637 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-podinfo" pod "b6ce7227-01e3-486c-93eb-f945903f32cf" (UID: "b6ce7227-01e3-486c-93eb-f945903f32cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:42:53.628371 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.628342 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:42:53.628371 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.628370 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:42:53.628589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.628383 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6ce7227-01e3-486c-93eb-f945903f32cf-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:42:53.628589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.628395 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqvf7\" (UniqueName: \"kubernetes.io/projected/b6ce7227-01e3-486c-93eb-f945903f32cf-kube-api-access-vqvf7\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:42:53.829355 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.829328 4603 generic.go:356] "Generic (PLEG): container finished" podID="b6ce7227-01e3-486c-93eb-f945903f32cf" containerID="a8718679c05bb9ac84c92c970d87ec8e374f27644d90aa3abe19df52bd623562" exitCode=0 Oct 30 18:42:53.829537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.829373 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" Oct 30 18:42:53.829537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.829413 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" event={"ID":"b6ce7227-01e3-486c-93eb-f945903f32cf","Type":"ContainerDied","Data":"a8718679c05bb9ac84c92c970d87ec8e374f27644d90aa3abe19df52bd623562"} Oct 30 18:42:53.829537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.829478 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv" event={"ID":"b6ce7227-01e3-486c-93eb-f945903f32cf","Type":"ContainerDied","Data":"a05705011d885672bd83c602b4d0a856ccf92092774362e3cec45420365f1843"} Oct 30 18:42:53.829537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.829531 4603 scope.go:122] "RemoveContainer" containerID="a8718679c05bb9ac84c92c970d87ec8e374f27644d90aa3abe19df52bd623562" Oct 30 18:42:53.831273 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.831246 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" event={"ID":"2bda071c-1825-4d2e-ab67-2a4acc90a8eb","Type":"ContainerStarted","Data":"c59330c073b3f02782d3b35b79751abf7efb43bfc9ff932eb007653620142add"} Oct 30 18:42:53.831384 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.831287 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" event={"ID":"2bda071c-1825-4d2e-ab67-2a4acc90a8eb","Type":"ContainerStarted","Data":"4937eff1fba3ccab0ff03884fe836e6d276888f5129442cd62f8da3228932eab"} Oct 30 18:42:53.839031 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.839015 4603 scope.go:122] "RemoveContainer" containerID="13f370e1c91d6b9858fa11610abd2d2b7a6e100840f20bcefa2e44363dac64be" Oct 30 18:42:53.847219 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.847201 4603 scope.go:122] "RemoveContainer" containerID="a8718679c05bb9ac84c92c970d87ec8e374f27644d90aa3abe19df52bd623562" Oct 30 18:42:53.847465 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:42:53.847446 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8718679c05bb9ac84c92c970d87ec8e374f27644d90aa3abe19df52bd623562\": container with ID starting with a8718679c05bb9ac84c92c970d87ec8e374f27644d90aa3abe19df52bd623562 not found: ID does not exist" containerID="a8718679c05bb9ac84c92c970d87ec8e374f27644d90aa3abe19df52bd623562" Oct 30 18:42:53.847586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.847477 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8718679c05bb9ac84c92c970d87ec8e374f27644d90aa3abe19df52bd623562"} err="failed to get container status \"a8718679c05bb9ac84c92c970d87ec8e374f27644d90aa3abe19df52bd623562\": rpc error: code = NotFound desc = could not find container \"a8718679c05bb9ac84c92c970d87ec8e374f27644d90aa3abe19df52bd623562\": container with ID starting with a8718679c05bb9ac84c92c970d87ec8e374f27644d90aa3abe19df52bd623562 not found: ID does not exist" Oct 30 18:42:53.847586 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.847517 4603 scope.go:122] "RemoveContainer" containerID="13f370e1c91d6b9858fa11610abd2d2b7a6e100840f20bcefa2e44363dac64be" Oct 30 18:42:53.847767 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:42:53.847750 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f370e1c91d6b9858fa11610abd2d2b7a6e100840f20bcefa2e44363dac64be\": container with ID starting with 13f370e1c91d6b9858fa11610abd2d2b7a6e100840f20bcefa2e44363dac64be not found: ID does not exist" containerID="13f370e1c91d6b9858fa11610abd2d2b7a6e100840f20bcefa2e44363dac64be" Oct 30 18:42:53.847850 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.847774 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f370e1c91d6b9858fa11610abd2d2b7a6e100840f20bcefa2e44363dac64be"} err="failed to get container status \"13f370e1c91d6b9858fa11610abd2d2b7a6e100840f20bcefa2e44363dac64be\": rpc error: code = NotFound desc = could not find container \"13f370e1c91d6b9858fa11610abd2d2b7a6e100840f20bcefa2e44363dac64be\": container with ID starting with 13f370e1c91d6b9858fa11610abd2d2b7a6e100840f20bcefa2e44363dac64be not found: ID does not exist" Oct 30 18:42:53.890771 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.890734 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" podStartSLOduration=1.890724478 podStartE2EDuration="1.890724478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:53.889322316 +0000 UTC m=+3034.604892189" watchObservedRunningTime="2026-10-30 18:42:53.890724478 +0000 UTC m=+3034.606294352" Oct 30 18:42:53.925258 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.925239 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv"] Oct 30 18:42:53.936809 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:53.936789 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-293234-openshift-default-7c756b857-hnkbv"] Oct 30 18:42:54.169288 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:54.169220 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:42:54.553992 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:54.553953 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:55.408021 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:55.407982 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6ce7227-01e3-486c-93eb-f945903f32cf" path="/var/lib/kubelet/pods/b6ce7227-01e3-486c-93eb-f945903f32cf/volumes" Oct 30 18:42:56.261187 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:56.261142 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:57.929387 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:57.929356 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:59.636725 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:42:59.636693 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:01.474933 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:01.474895 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:03.158463 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:03.158428 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:03.169756 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:03.169724 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:43:03.184909 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:03.184882 4603 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:43:03.896710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:03.896683 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:43:04.842349 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:04.842311 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:06.435040 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:06.435003 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:08.147025 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:08.146985 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:10.007339 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:10.007309 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:11.655147 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:11.655113 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:13.247871 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:13.247817 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:14.966528 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:14.962454 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:16.657194 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:16.657157 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:18.281927 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:18.281887 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:18.606557 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:18.606436 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn"] Oct 30 18:43:18.678191 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:18.678157 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:18.796516 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:18.796463 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5"] Oct 30 18:43:18.796743 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:18.796719 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" podUID="0d5bcd9f-9b13-40b9-9922-dadc505730d3" containerName="istio-proxy" containerID="cri-o://3efe6cc562c99701dc3967cbe3cbc53fb0a306cd6c21e7e294c9d5b8199f5f95" gracePeriod=30 Oct 30 18:43:18.876219 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:18.876126 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:19.025156 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:19.025124 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:19.226558 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:19.226451 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:19.417252 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:19.417223 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:19.723483 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:19.723447 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:20.064396 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:20.064352 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:21.640511 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:21.640469 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:21.821368 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:21.821330 4603 patch_prober.go:29] interesting pod/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:43:21.821644 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:21.821398 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" podUID="0d5bcd9f-9b13-40b9-9922-dadc505730d3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:43:21.930349 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:43:21.930271 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:43:23.264260 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:23.264221 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:24.022489 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.022458 4603 generic.go:356] "Generic (PLEG): container finished" podID="0d5bcd9f-9b13-40b9-9922-dadc505730d3" containerID="3efe6cc562c99701dc3967cbe3cbc53fb0a306cd6c21e7e294c9d5b8199f5f95" exitCode=0 Oct 30 18:43:24.022843 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.022820 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" event={"ID":"0d5bcd9f-9b13-40b9-9922-dadc505730d3","Type":"ContainerDied","Data":"3efe6cc562c99701dc3967cbe3cbc53fb0a306cd6c21e7e294c9d5b8199f5f95"} Oct 30 18:43:24.133632 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:43:24.133542 4603 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5bcd9f_9b13_40b9_9922_dadc505730d3.slice/crio-conmon-3efe6cc562c99701dc3967cbe3cbc53fb0a306cd6c21e7e294c9d5b8199f5f95.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5bcd9f_9b13_40b9_9922_dadc505730d3.slice/crio-3efe6cc562c99701dc3967cbe3cbc53fb0a306cd6c21e7e294c9d5b8199f5f95.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:43:24.561529 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.557155 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:24.737766 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.737739 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:43:24.829041 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.828955 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-workload-certs\") pod \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " Oct 30 18:43:24.829224 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.829057 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istiod-ca-cert\") pod \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " Oct 30 18:43:24.829224 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.829117 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d5bcd9f-9b13-40b9-9922-dadc505730d3-kube-api-access-dg8jx\" (UniqueName: \"kubernetes.io/projected/0d5bcd9f-9b13-40b9-9922-dadc505730d3-kube-api-access-dg8jx\") pod \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " Oct 30 18:43:24.829224 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.829167 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-envoy\") pod \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " Oct 30 18:43:24.829224 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.829205 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-podinfo\") pod \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " Oct 30 18:43:24.829478 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.829259 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-token\" (UniqueName: \"kubernetes.io/projected/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-token\") pod \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " Oct 30 18:43:24.829478 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.829260 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-workload-certs" pod "0d5bcd9f-9b13-40b9-9922-dadc505730d3" (UID: "0d5bcd9f-9b13-40b9-9922-dadc505730d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:24.829478 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.829292 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-credential-socket\") pod \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " Oct 30 18:43:24.829478 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.829336 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-data\") pod \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " Oct 30 18:43:24.829478 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.829407 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-workload-socket\") pod \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\" (UID: \"0d5bcd9f-9b13-40b9-9922-dadc505730d3\") " Oct 30 18:43:24.829814 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.829559 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istiod-ca-cert" pod "0d5bcd9f-9b13-40b9-9922-dadc505730d3" (UID: "0d5bcd9f-9b13-40b9-9922-dadc505730d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:24.829871 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.829822 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:43:24.829871 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.829843 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:43:24.829979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.829860 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-credential-socket" pod "0d5bcd9f-9b13-40b9-9922-dadc505730d3" (UID: "0d5bcd9f-9b13-40b9-9922-dadc505730d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:24.829979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.829951 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-data" pod "0d5bcd9f-9b13-40b9-9922-dadc505730d3" (UID: "0d5bcd9f-9b13-40b9-9922-dadc505730d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:24.830271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.830046 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-workload-socket" pod "0d5bcd9f-9b13-40b9-9922-dadc505730d3" (UID: "0d5bcd9f-9b13-40b9-9922-dadc505730d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:24.832351 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.832296 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-token" pod "0d5bcd9f-9b13-40b9-9922-dadc505730d3" (UID: "0d5bcd9f-9b13-40b9-9922-dadc505730d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:24.832351 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.832307 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-podinfo" pod "0d5bcd9f-9b13-40b9-9922-dadc505730d3" (UID: "0d5bcd9f-9b13-40b9-9922-dadc505730d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:43:24.833247 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.833226 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-envoy" pod "0d5bcd9f-9b13-40b9-9922-dadc505730d3" (UID: "0d5bcd9f-9b13-40b9-9922-dadc505730d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:24.833370 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.833267 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5bcd9f-9b13-40b9-9922-dadc505730d3-kube-api-access-dg8jx" pod "0d5bcd9f-9b13-40b9-9922-dadc505730d3" (UID: "0d5bcd9f-9b13-40b9-9922-dadc505730d3"). InnerVolumeSpecName "kube-api-access-dg8jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:24.930810 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.930785 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg8jx\" (UniqueName: \"kubernetes.io/projected/0d5bcd9f-9b13-40b9-9922-dadc505730d3-kube-api-access-dg8jx\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:43:24.930931 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.930819 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:43:24.930931 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.930829 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:43:24.930931 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.930837 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:43:24.930931 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.930846 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:43:24.930931 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.930853 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:43:24.930931 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:24.930862 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5bcd9f-9b13-40b9-9922-dadc505730d3-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:43:25.029099 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:25.029074 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" Oct 30 18:43:25.029216 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:25.029116 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5" event={"ID":"0d5bcd9f-9b13-40b9-9922-dadc505730d3","Type":"ContainerDied","Data":"f5d116cd5310200b619805f95ad1150c031cd10e6519b4fe95fd08eed6dd9e38"} Oct 30 18:43:25.029216 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:25.029176 4603 scope.go:122] "RemoveContainer" containerID="3efe6cc562c99701dc3967cbe3cbc53fb0a306cd6c21e7e294c9d5b8199f5f95" Oct 30 18:43:25.041852 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:25.040469 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:25.084196 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:25.084097 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5"] Oct 30 18:43:25.091179 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:25.091151 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-366d9f-openshift-default-5f56b7785c-hk2g5"] Oct 30 18:43:25.408561 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:25.408474 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d5bcd9f-9b13-40b9-9922-dadc505730d3" path="/var/lib/kubelet/pods/0d5bcd9f-9b13-40b9-9922-dadc505730d3/volumes" Oct 30 18:43:26.656513 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:26.656461 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:28.450380 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:28.450035 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:30.151538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:30.150828 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:31.860485 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:31.860445 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:33.627979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:33.627916 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" podUID="2bda071c-1825-4d2e-ab67-2a4acc90a8eb" containerName="main" containerID="cri-o://c59330c073b3f02782d3b35b79751abf7efb43bfc9ff932eb007653620142add" gracePeriod=45 Oct 30 18:43:33.656158 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:33.656132 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:33.890701 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:33.890649 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:43:33.945610 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:33.945588 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-tmp-dir\") pod \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " Oct 30 18:43:33.945745 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:33.945650 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-home\" (UniqueName: \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-home\") pod \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " Oct 30 18:43:33.945745 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:33.945683 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-kube-api-access-l6qhq\" (UniqueName: \"kubernetes.io/projected/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-kube-api-access-l6qhq\") pod \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " Oct 30 18:43:33.945745 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:33.945708 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-dshm\") pod \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " Oct 30 18:43:33.945932 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:33.945745 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-model-cache\") pod \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " Oct 30 18:43:33.945932 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:33.945822 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-tls-certs\" (UniqueName: \"kubernetes.io/secret/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-tls-certs\") pod \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\" (UID: \"2bda071c-1825-4d2e-ab67-2a4acc90a8eb\") " Oct 30 18:43:33.946041 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:33.945926 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-tmp-dir" pod "2bda071c-1825-4d2e-ab67-2a4acc90a8eb" (UID: "2bda071c-1825-4d2e-ab67-2a4acc90a8eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:33.946041 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:33.945975 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-home" pod "2bda071c-1825-4d2e-ab67-2a4acc90a8eb" (UID: "2bda071c-1825-4d2e-ab67-2a4acc90a8eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:33.946175 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:33.946149 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-model-cache" pod "2bda071c-1825-4d2e-ab67-2a4acc90a8eb" (UID: "2bda071c-1825-4d2e-ab67-2a4acc90a8eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:33.946300 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:33.946282 4603 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-tmp-dir\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:43:33.946362 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:33.946301 4603 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-home\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:43:33.946362 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:33.946310 4603 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-model-cache\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:43:33.948699 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:33.948672 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-dshm" pod "2bda071c-1825-4d2e-ab67-2a4acc90a8eb" (UID: "2bda071c-1825-4d2e-ab67-2a4acc90a8eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:33.948814 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:33.948739 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-tls-certs" pod "2bda071c-1825-4d2e-ab67-2a4acc90a8eb" (UID: "2bda071c-1825-4d2e-ab67-2a4acc90a8eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:43:33.948814 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:33.948756 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-kube-api-access-l6qhq" pod "2bda071c-1825-4d2e-ab67-2a4acc90a8eb" (UID: "2bda071c-1825-4d2e-ab67-2a4acc90a8eb"). InnerVolumeSpecName "kube-api-access-l6qhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:34.047599 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:34.047569 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6qhq\" (UniqueName: \"kubernetes.io/projected/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-kube-api-access-l6qhq\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:43:34.047599 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:34.047597 4603 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-dshm\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:43:34.047747 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:34.047612 4603 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bda071c-1825-4d2e-ab67-2a4acc90a8eb-tls-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:43:34.140106 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:34.140084 4603 generic.go:356] "Generic (PLEG): container finished" podID="2bda071c-1825-4d2e-ab67-2a4acc90a8eb" containerID="c59330c073b3f02782d3b35b79751abf7efb43bfc9ff932eb007653620142add" exitCode=0 Oct 30 18:43:34.140209 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:34.140124 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" Oct 30 18:43:34.140209 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:34.140165 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" event={"ID":"2bda071c-1825-4d2e-ab67-2a4acc90a8eb","Type":"ContainerDied","Data":"c59330c073b3f02782d3b35b79751abf7efb43bfc9ff932eb007653620142add"} Oct 30 18:43:34.140209 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:34.140206 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" event={"ID":"2bda071c-1825-4d2e-ab67-2a4acc90a8eb","Type":"ContainerDied","Data":"4937eff1fba3ccab0ff03884fe836e6d276888f5129442cd62f8da3228932eab"} Oct 30 18:43:34.140330 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:34.140224 4603 scope.go:122] "RemoveContainer" containerID="c59330c073b3f02782d3b35b79751abf7efb43bfc9ff932eb007653620142add" Oct 30 18:43:34.149669 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:34.149653 4603 scope.go:122] "RemoveContainer" containerID="c59330c073b3f02782d3b35b79751abf7efb43bfc9ff932eb007653620142add" Oct 30 18:43:34.149963 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:43:34.149944 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c59330c073b3f02782d3b35b79751abf7efb43bfc9ff932eb007653620142add\": container with ID starting with c59330c073b3f02782d3b35b79751abf7efb43bfc9ff932eb007653620142add not found: ID does not exist" containerID="c59330c073b3f02782d3b35b79751abf7efb43bfc9ff932eb007653620142add" Oct 30 18:43:34.150015 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:34.149971 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c59330c073b3f02782d3b35b79751abf7efb43bfc9ff932eb007653620142add"} err="failed to get container status \"c59330c073b3f02782d3b35b79751abf7efb43bfc9ff932eb007653620142add\": rpc error: code = NotFound desc = could not find container \"c59330c073b3f02782d3b35b79751abf7efb43bfc9ff932eb007653620142add\": container with ID starting with c59330c073b3f02782d3b35b79751abf7efb43bfc9ff932eb007653620142add not found: ID does not exist" Oct 30 18:43:34.189866 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:34.189841 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn"] Oct 30 18:43:34.193933 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:34.193911 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn"] Oct 30 18:43:34.887266 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:34.887223 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-noten-366d9f/noten-model-366d9f-kserve-76484d4f8f-l7srn" podUID="2bda071c-1825-4d2e-ab67-2a4acc90a8eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.180:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:43:35.240610 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:35.240492 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:35.408810 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:35.408782 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bda071c-1825-4d2e-ab67-2a4acc90a8eb" path="/var/lib/kubelet/pods/2bda071c-1825-4d2e-ab67-2a4acc90a8eb/volumes" Oct 30 18:43:36.842748 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:36.842715 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:38.672227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:38.672180 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:40.340433 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:40.340391 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:41.954626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:41.954561 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:43.740393 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:43.740350 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:45.347846 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:45.347810 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:47.146413 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:47.146382 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:48.938865 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:48.938828 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:50.865144 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:50.865098 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:51.930631 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:43:51.930598 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:43:52.554864 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:52.554836 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:54.403094 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:54.403058 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:56.142182 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:56.142154 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:57.740461 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:57.740427 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:59.481530 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:43:59.473080 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:01.361968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:01.361932 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:01.523714 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:01.523679 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:01.705387 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:01.705301 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:03.234491 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:03.234396 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:04.847712 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:04.847668 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:05.027540 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:05.027494 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:06.642889 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:06.642859 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc"] Oct 30 18:44:06.655538 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:06.652727 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf"] Oct 30 18:44:06.663395 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:06.663363 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv"] Oct 30 18:44:06.828731 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:06.828690 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:07.000579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:06.997585 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:08.716961 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:08.716928 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:10.316055 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:10.316010 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:12.082078 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:12.082018 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:12.841252 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:12.841223 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:13.017108 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:13.017076 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:13.848745 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:13.848713 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:15.253428 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.253391 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d20c55c3-mxp4f"] Oct 30 18:44:15.253883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.253835 4603 cpu_manager.go:414] "containerMap: removing container" podUID="2bda071c-1825-4d2e-ab67-2a4acc90a8eb" containerName="main" Oct 30 18:44:15.253883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.253851 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bda071c-1825-4d2e-ab67-2a4acc90a8eb" containerName="main" Oct 30 18:44:15.253883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.253871 4603 cpu_manager.go:414] "containerMap: removing container" podUID="b6ce7227-01e3-486c-93eb-f945903f32cf" containerName="istio-proxy" Oct 30 18:44:15.253883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.253876 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6ce7227-01e3-486c-93eb-f945903f32cf" containerName="istio-proxy" Oct 30 18:44:15.253883 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.253885 4603 cpu_manager.go:414] "containerMap: removing container" podUID="b6ce7227-01e3-486c-93eb-f945903f32cf" containerName="istio-proxy" Oct 30 18:44:15.254073 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.253891 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6ce7227-01e3-486c-93eb-f945903f32cf" containerName="istio-proxy" Oct 30 18:44:15.254073 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.253900 4603 cpu_manager.go:414] "containerMap: removing container" podUID="0d5bcd9f-9b13-40b9-9922-dadc505730d3" containerName="istio-proxy" Oct 30 18:44:15.254073 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.253906 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d5bcd9f-9b13-40b9-9922-dadc505730d3" containerName="istio-proxy" Oct 30 18:44:15.254073 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.253966 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bda071c-1825-4d2e-ab67-2a4acc90a8eb" containerName="main" Oct 30 18:44:15.254073 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.253975 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6ce7227-01e3-486c-93eb-f945903f32cf" containerName="istio-proxy" Oct 30 18:44:15.254073 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.253981 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6ce7227-01e3-486c-93eb-f945903f32cf" containerName="istio-proxy" Oct 30 18:44:15.254073 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.253988 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d5bcd9f-9b13-40b9-9922-dadc505730d3" containerName="istio-proxy" Oct 30 18:44:15.257210 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.257184 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20c55c3-mxp4f" Oct 30 18:44:15.279093 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.279068 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d20c55c3-mxp4f"] Oct 30 18:44:15.284026 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.284004 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93bc70e7-e90f-4b6e-886b-9bbbb126d578-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d20c55c3-mxp4f\" (UID: \"93bc70e7-e90f-4b6e-886b-9bbbb126d578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20c55c3-mxp4f" Oct 30 18:44:15.284173 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.284063 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6xrk\" (UniqueName: \"kubernetes.io/projected/93bc70e7-e90f-4b6e-886b-9bbbb126d578-kube-api-access-v6xrk\") pod \"maas-api-revoke-sub-d20c55c3-mxp4f\" (UID: \"93bc70e7-e90f-4b6e-886b-9bbbb126d578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20c55c3-mxp4f" Oct 30 18:44:15.284244 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.284199 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93bc70e7-e90f-4b6e-886b-9bbbb126d578-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d20c55c3-mxp4f\" (UID: \"93bc70e7-e90f-4b6e-886b-9bbbb126d578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20c55c3-mxp4f" Oct 30 18:44:15.355606 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.355582 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:44:15.355718 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.355606 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:44:15.385319 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.385299 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93bc70e7-e90f-4b6e-886b-9bbbb126d578-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d20c55c3-mxp4f\" (UID: \"93bc70e7-e90f-4b6e-886b-9bbbb126d578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20c55c3-mxp4f" Oct 30 18:44:15.385403 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.385339 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93bc70e7-e90f-4b6e-886b-9bbbb126d578-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d20c55c3-mxp4f\" (UID: \"93bc70e7-e90f-4b6e-886b-9bbbb126d578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20c55c3-mxp4f" Oct 30 18:44:15.385454 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.385401 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6xrk\" (UniqueName: \"kubernetes.io/projected/93bc70e7-e90f-4b6e-886b-9bbbb126d578-kube-api-access-v6xrk\") pod \"maas-api-revoke-sub-d20c55c3-mxp4f\" (UID: \"93bc70e7-e90f-4b6e-886b-9bbbb126d578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20c55c3-mxp4f" Oct 30 18:44:15.386162 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.386091 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93bc70e7-e90f-4b6e-886b-9bbbb126d578-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d20c55c3-mxp4f\" (UID: \"93bc70e7-e90f-4b6e-886b-9bbbb126d578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20c55c3-mxp4f" Oct 30 18:44:15.387803 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.387783 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93bc70e7-e90f-4b6e-886b-9bbbb126d578-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d20c55c3-mxp4f\" (UID: \"93bc70e7-e90f-4b6e-886b-9bbbb126d578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20c55c3-mxp4f" Oct 30 18:44:15.409281 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.409257 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6xrk\" (UniqueName: \"kubernetes.io/projected/93bc70e7-e90f-4b6e-886b-9bbbb126d578-kube-api-access-v6xrk\") pod \"maas-api-revoke-sub-d20c55c3-mxp4f\" (UID: \"93bc70e7-e90f-4b6e-886b-9bbbb126d578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20c55c3-mxp4f" Oct 30 18:44:15.570952 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.570920 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20c55c3-mxp4f" Oct 30 18:44:15.875424 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:15.875397 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d20c55c3-mxp4f"] Oct 30 18:44:15.877846 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:44:15.877813 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93bc70e7_e90f_4b6e_886b_9bbbb126d578.slice/crio-cecebeefc44f1773cd6455ff02482bdd7b029c4018cd48ad92ea063c7d4c3100 WatchSource:0}: Error finding container cecebeefc44f1773cd6455ff02482bdd7b029c4018cd48ad92ea063c7d4c3100: Status 404 returned error can't find the container with id cecebeefc44f1773cd6455ff02482bdd7b029c4018cd48ad92ea063c7d4c3100 Oct 30 18:44:16.426772 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:16.426744 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20c55c3-mxp4f" event={"ID":"93bc70e7-e90f-4b6e-886b-9bbbb126d578","Type":"ContainerStarted","Data":"7b4a658dddbed94eeb55aace682773242f7cdd636008b43f6a53b50c8e339886"} Oct 30 18:44:16.427142 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:16.426788 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20c55c3-mxp4f" event={"ID":"93bc70e7-e90f-4b6e-886b-9bbbb126d578","Type":"ContainerStarted","Data":"cecebeefc44f1773cd6455ff02482bdd7b029c4018cd48ad92ea063c7d4c3100"} Oct 30 18:44:16.466624 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:16.466541 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20c55c3-mxp4f" podStartSLOduration=1.46652353 podStartE2EDuration="1.46652353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:16.465789354 +0000 UTC m=+3117.181359228" watchObservedRunningTime="2026-10-30 18:44:16.46652353 +0000 UTC m=+3117.182093395" Oct 30 18:44:17.435285 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:17.435250 4603 generic.go:356] "Generic (PLEG): container finished" podID="93bc70e7-e90f-4b6e-886b-9bbbb126d578" containerID="7b4a658dddbed94eeb55aace682773242f7cdd636008b43f6a53b50c8e339886" exitCode=0 Oct 30 18:44:17.435995 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:17.435434 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20c55c3-mxp4f" event={"ID":"93bc70e7-e90f-4b6e-886b-9bbbb126d578","Type":"ContainerDied","Data":"7b4a658dddbed94eeb55aace682773242f7cdd636008b43f6a53b50c8e339886"} Oct 30 18:44:19.016110 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:19.016087 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20c55c3-mxp4f" Oct 30 18:44:19.086156 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:19.086122 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93bc70e7-e90f-4b6e-886b-9bbbb126d578-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93bc70e7-e90f-4b6e-886b-9bbbb126d578-maas-api-openshift-ca\") pod \"93bc70e7-e90f-4b6e-886b-9bbbb126d578\" (UID: \"93bc70e7-e90f-4b6e-886b-9bbbb126d578\") " Oct 30 18:44:19.086315 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:19.086240 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93bc70e7-e90f-4b6e-886b-9bbbb126d578-kube-api-access-v6xrk\" (UniqueName: \"kubernetes.io/projected/93bc70e7-e90f-4b6e-886b-9bbbb126d578-kube-api-access-v6xrk\") pod \"93bc70e7-e90f-4b6e-886b-9bbbb126d578\" (UID: \"93bc70e7-e90f-4b6e-886b-9bbbb126d578\") " Oct 30 18:44:19.086315 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:19.086291 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93bc70e7-e90f-4b6e-886b-9bbbb126d578-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93bc70e7-e90f-4b6e-886b-9bbbb126d578-maas-api-xks-ca\") pod \"93bc70e7-e90f-4b6e-886b-9bbbb126d578\" (UID: \"93bc70e7-e90f-4b6e-886b-9bbbb126d578\") " Oct 30 18:44:19.086518 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:19.086475 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93bc70e7-e90f-4b6e-886b-9bbbb126d578-maas-api-openshift-ca" pod "93bc70e7-e90f-4b6e-886b-9bbbb126d578" (UID: "93bc70e7-e90f-4b6e-886b-9bbbb126d578"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:19.086623 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:19.086603 4603 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93bc70e7-e90f-4b6e-886b-9bbbb126d578-maas-api-openshift-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:19.088497 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:19.088467 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93bc70e7-e90f-4b6e-886b-9bbbb126d578-kube-api-access-v6xrk" pod "93bc70e7-e90f-4b6e-886b-9bbbb126d578" (UID: "93bc70e7-e90f-4b6e-886b-9bbbb126d578"). InnerVolumeSpecName "kube-api-access-v6xrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:19.088618 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:19.088597 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93bc70e7-e90f-4b6e-886b-9bbbb126d578-maas-api-xks-ca" pod "93bc70e7-e90f-4b6e-886b-9bbbb126d578" (UID: "93bc70e7-e90f-4b6e-886b-9bbbb126d578"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:19.154320 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:19.154257 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:19.187443 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:19.187419 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6xrk\" (UniqueName: \"kubernetes.io/projected/93bc70e7-e90f-4b6e-886b-9bbbb126d578-kube-api-access-v6xrk\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:19.187443 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:19.187443 4603 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93bc70e7-e90f-4b6e-886b-9bbbb126d578-maas-api-xks-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:19.314877 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:19.314841 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:19.490710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:19.490683 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20c55c3-mxp4f" Oct 30 18:44:19.490850 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:19.490708 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20c55c3-mxp4f" event={"ID":"93bc70e7-e90f-4b6e-886b-9bbbb126d578","Type":"ContainerDied","Data":"cecebeefc44f1773cd6455ff02482bdd7b029c4018cd48ad92ea063c7d4c3100"} Oct 30 18:44:19.490850 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:19.490756 4603 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cecebeefc44f1773cd6455ff02482bdd7b029c4018cd48ad92ea063c7d4c3100" Oct 30 18:44:21.848778 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:21.848739 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" containerID="cri-o://5cb353540add350c5f2dbf1340485e8b8652709eb34c434a65ed2f2a1fb277ba" gracePeriod=45 Oct 30 18:44:21.896977 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:21.896930 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" containerID="cri-o://c588e962ba8430936ef1325b3f47796b8318e25ee6fa15594678d697d6bedfe1" gracePeriod=45 Oct 30 18:44:21.932395 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:44:21.932367 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:44:22.073098 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.071831 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" containerID="cri-o://f0f518fa4ff68de6a8fd17fc6c8e60bff8745c593d4c0669319e72ec6fdaeeaa" gracePeriod=45 Oct 30 18:44:22.197845 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.197818 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:44:22.201456 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.201436 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:44:22.313784 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.313746 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb34381d-3c03-4aa1-9af1-7d000b687af0-kube-api-access-nnzp6\" (UniqueName: \"kubernetes.io/projected/bb34381d-3c03-4aa1-9af1-7d000b687af0-kube-api-access-nnzp6\") pod \"bb34381d-3c03-4aa1-9af1-7d000b687af0\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " Oct 30 18:44:22.313979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.313813 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-tls-certs\") pod \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " Oct 30 18:44:22.313979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.313869 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-home\" (UniqueName: \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-home\") pod \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " Oct 30 18:44:22.313979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.313920 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-model-cache\") pod \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " Oct 30 18:44:22.313979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.313962 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-tmp-dir\") pod \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " Oct 30 18:44:22.314230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.314004 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-home\" (UniqueName: \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-home\") pod \"bb34381d-3c03-4aa1-9af1-7d000b687af0\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " Oct 30 18:44:22.314230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.314037 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-dshm\") pod \"bb34381d-3c03-4aa1-9af1-7d000b687af0\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " Oct 30 18:44:22.314230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.314072 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb34381d-3c03-4aa1-9af1-7d000b687af0-tls-certs\" (UniqueName: \"kubernetes.io/secret/bb34381d-3c03-4aa1-9af1-7d000b687af0-tls-certs\") pod \"bb34381d-3c03-4aa1-9af1-7d000b687af0\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " Oct 30 18:44:22.314230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.314124 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-tmp-dir\") pod \"bb34381d-3c03-4aa1-9af1-7d000b687af0\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " Oct 30 18:44:22.314230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.314152 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-home" pod "e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" (UID: "e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:22.314230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.314179 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-kube-api-access-2lccs\" (UniqueName: \"kubernetes.io/projected/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-kube-api-access-2lccs\") pod \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " Oct 30 18:44:22.314230 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.314225 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-model-cache\") pod \"bb34381d-3c03-4aa1-9af1-7d000b687af0\" (UID: \"bb34381d-3c03-4aa1-9af1-7d000b687af0\") " Oct 30 18:44:22.314646 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.314263 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-dshm\") pod \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\" (UID: \"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9\") " Oct 30 18:44:22.314646 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.314371 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-home" pod "bb34381d-3c03-4aa1-9af1-7d000b687af0" (UID: "bb34381d-3c03-4aa1-9af1-7d000b687af0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:22.314646 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.314375 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-model-cache" pod "e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" (UID: "e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:22.314646 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.314578 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-tmp-dir" pod "bb34381d-3c03-4aa1-9af1-7d000b687af0" (UID: "bb34381d-3c03-4aa1-9af1-7d000b687af0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:22.314873 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.314727 4603 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-tmp-dir\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.314873 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.314752 4603 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-home\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.314873 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.314768 4603 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-model-cache\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.314873 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.314782 4603 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-home\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.315534 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.315478 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-model-cache" pod "bb34381d-3c03-4aa1-9af1-7d000b687af0" (UID: "bb34381d-3c03-4aa1-9af1-7d000b687af0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:22.315679 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.315587 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-tmp-dir" pod "e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" (UID: "e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:22.316890 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.316861 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-tls-certs" pod "e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" (UID: "e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:22.317231 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.317208 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-dshm" pod "e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" (UID: "e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:22.317926 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.317893 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb34381d-3c03-4aa1-9af1-7d000b687af0-tls-certs" pod "bb34381d-3c03-4aa1-9af1-7d000b687af0" (UID: "bb34381d-3c03-4aa1-9af1-7d000b687af0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:22.318076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.318045 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-dshm" pod "bb34381d-3c03-4aa1-9af1-7d000b687af0" (UID: "bb34381d-3c03-4aa1-9af1-7d000b687af0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:22.318160 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.318080 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-kube-api-access-2lccs" pod "e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" (UID: "e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9"). InnerVolumeSpecName "kube-api-access-2lccs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:22.318921 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.318820 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb34381d-3c03-4aa1-9af1-7d000b687af0-kube-api-access-nnzp6" pod "bb34381d-3c03-4aa1-9af1-7d000b687af0" (UID: "bb34381d-3c03-4aa1-9af1-7d000b687af0"). InnerVolumeSpecName "kube-api-access-nnzp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:22.334600 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.334581 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:44:22.415765 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.415692 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-tls-certs\") pod \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " Oct 30 18:44:22.415931 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.415790 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-model-cache\") pod \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " Oct 30 18:44:22.415931 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.415849 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-dshm\") pod \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " Oct 30 18:44:22.415931 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.415891 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-tmp-dir\") pod \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " Oct 30 18:44:22.416120 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.415977 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-kube-api-access-dkk9w\" (UniqueName: \"kubernetes.io/projected/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-kube-api-access-dkk9w\") pod \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " Oct 30 18:44:22.416120 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.416012 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-home\" (UniqueName: \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-home\") pod \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\" (UID: \"ea5f82db-b4a7-4c43-8131-dcffe2baa65b\") " Oct 30 18:44:22.416120 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.416093 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-model-cache" pod "ea5f82db-b4a7-4c43-8131-dcffe2baa65b" (UID: "ea5f82db-b4a7-4c43-8131-dcffe2baa65b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:22.416259 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.416142 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-tmp-dir" pod "ea5f82db-b4a7-4c43-8131-dcffe2baa65b" (UID: "ea5f82db-b4a7-4c43-8131-dcffe2baa65b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:22.416399 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.416359 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-home" pod "ea5f82db-b4a7-4c43-8131-dcffe2baa65b" (UID: "ea5f82db-b4a7-4c43-8131-dcffe2baa65b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:22.416482 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.416433 4603 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-tmp-dir\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.416482 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.416452 4603 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-tmp-dir\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.416482 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.416465 4603 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-dshm\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.416482 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.416473 4603 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb34381d-3c03-4aa1-9af1-7d000b687af0-tls-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.416482 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.416481 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lccs\" (UniqueName: \"kubernetes.io/projected/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-kube-api-access-2lccs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.416949 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.416491 4603 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb34381d-3c03-4aa1-9af1-7d000b687af0-model-cache\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.416949 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.416533 4603 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-dshm\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.416949 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.416547 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnzp6\" (UniqueName: \"kubernetes.io/projected/bb34381d-3c03-4aa1-9af1-7d000b687af0-kube-api-access-nnzp6\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.416949 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.416557 4603 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9-tls-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.416949 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.416565 4603 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-model-cache\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.418275 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.418247 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-kube-api-access-dkk9w" pod "ea5f82db-b4a7-4c43-8131-dcffe2baa65b" (UID: "ea5f82db-b4a7-4c43-8131-dcffe2baa65b"). InnerVolumeSpecName "kube-api-access-dkk9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:22.418386 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.418328 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-tls-certs" pod "ea5f82db-b4a7-4c43-8131-dcffe2baa65b" (UID: "ea5f82db-b4a7-4c43-8131-dcffe2baa65b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:22.418536 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.418513 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-dshm" pod "ea5f82db-b4a7-4c43-8131-dcffe2baa65b" (UID: "ea5f82db-b4a7-4c43-8131-dcffe2baa65b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:22.506865 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.506827 4603 generic.go:356] "Generic (PLEG): container finished" podID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerID="c588e962ba8430936ef1325b3f47796b8318e25ee6fa15594678d697d6bedfe1" exitCode=0 Oct 30 18:44:22.507060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.506872 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" Oct 30 18:44:22.507060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.506920 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" event={"ID":"bb34381d-3c03-4aa1-9af1-7d000b687af0","Type":"ContainerDied","Data":"c588e962ba8430936ef1325b3f47796b8318e25ee6fa15594678d697d6bedfe1"} Oct 30 18:44:22.507060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.506973 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc" event={"ID":"bb34381d-3c03-4aa1-9af1-7d000b687af0","Type":"ContainerDied","Data":"bfc2a551d2fad09e8441f6393bb28f8b0014fc9f07f471a537980964960196c6"} Oct 30 18:44:22.507060 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.506998 4603 scope.go:122] "RemoveContainer" containerID="c588e962ba8430936ef1325b3f47796b8318e25ee6fa15594678d697d6bedfe1" Oct 30 18:44:22.508293 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.508266 4603 generic.go:356] "Generic (PLEG): container finished" podID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerID="f0f518fa4ff68de6a8fd17fc6c8e60bff8745c593d4c0669319e72ec6fdaeeaa" exitCode=0 Oct 30 18:44:22.508400 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.508329 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" Oct 30 18:44:22.508400 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.508368 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" event={"ID":"ea5f82db-b4a7-4c43-8131-dcffe2baa65b","Type":"ContainerDied","Data":"f0f518fa4ff68de6a8fd17fc6c8e60bff8745c593d4c0669319e72ec6fdaeeaa"} Oct 30 18:44:22.508524 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.508405 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv" event={"ID":"ea5f82db-b4a7-4c43-8131-dcffe2baa65b","Type":"ContainerDied","Data":"e24ffe5d7ffda84cbfc3145069cfb606f164d0f532fe1e7f8ed35bf9eed205b6"} Oct 30 18:44:22.510022 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.509999 4603 generic.go:356] "Generic (PLEG): container finished" podID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerID="5cb353540add350c5f2dbf1340485e8b8652709eb34c434a65ed2f2a1fb277ba" exitCode=0 Oct 30 18:44:22.510152 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.510039 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" event={"ID":"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9","Type":"ContainerDied","Data":"5cb353540add350c5f2dbf1340485e8b8652709eb34c434a65ed2f2a1fb277ba"} Oct 30 18:44:22.510152 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.510090 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" event={"ID":"e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9","Type":"ContainerDied","Data":"17d926044da064d3f3edbec0dd54efaa09c1554cd0e03167bfa03df1815fc7ca"} Oct 30 18:44:22.510152 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.510054 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf" Oct 30 18:44:22.517700 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.517628 4603 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-dshm\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.517700 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.517650 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkk9w\" (UniqueName: \"kubernetes.io/projected/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-kube-api-access-dkk9w\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.517700 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.517665 4603 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-home\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.517700 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.517678 4603 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5f82db-b4a7-4c43-8131-dcffe2baa65b-tls-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.517954 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.517731 4603 scope.go:122] "RemoveContainer" containerID="c588e962ba8430936ef1325b3f47796b8318e25ee6fa15594678d697d6bedfe1" Oct 30 18:44:22.518075 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:44:22.518044 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c588e962ba8430936ef1325b3f47796b8318e25ee6fa15594678d697d6bedfe1\": container with ID starting with c588e962ba8430936ef1325b3f47796b8318e25ee6fa15594678d697d6bedfe1 not found: ID does not exist" containerID="c588e962ba8430936ef1325b3f47796b8318e25ee6fa15594678d697d6bedfe1" Oct 30 18:44:22.518140 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.518088 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c588e962ba8430936ef1325b3f47796b8318e25ee6fa15594678d697d6bedfe1"} err="failed to get container status \"c588e962ba8430936ef1325b3f47796b8318e25ee6fa15594678d697d6bedfe1\": rpc error: code = NotFound desc = could not find container \"c588e962ba8430936ef1325b3f47796b8318e25ee6fa15594678d697d6bedfe1\": container with ID starting with c588e962ba8430936ef1325b3f47796b8318e25ee6fa15594678d697d6bedfe1 not found: ID does not exist" Oct 30 18:44:22.518140 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.518112 4603 scope.go:122] "RemoveContainer" containerID="f0f518fa4ff68de6a8fd17fc6c8e60bff8745c593d4c0669319e72ec6fdaeeaa" Oct 30 18:44:22.526855 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.526837 4603 scope.go:122] "RemoveContainer" containerID="f0f518fa4ff68de6a8fd17fc6c8e60bff8745c593d4c0669319e72ec6fdaeeaa" Oct 30 18:44:22.527113 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:44:22.527092 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f518fa4ff68de6a8fd17fc6c8e60bff8745c593d4c0669319e72ec6fdaeeaa\": container with ID starting with f0f518fa4ff68de6a8fd17fc6c8e60bff8745c593d4c0669319e72ec6fdaeeaa not found: ID does not exist" containerID="f0f518fa4ff68de6a8fd17fc6c8e60bff8745c593d4c0669319e72ec6fdaeeaa" Oct 30 18:44:22.527204 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.527123 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f518fa4ff68de6a8fd17fc6c8e60bff8745c593d4c0669319e72ec6fdaeeaa"} err="failed to get container status \"f0f518fa4ff68de6a8fd17fc6c8e60bff8745c593d4c0669319e72ec6fdaeeaa\": rpc error: code = NotFound desc = could not find container \"f0f518fa4ff68de6a8fd17fc6c8e60bff8745c593d4c0669319e72ec6fdaeeaa\": container with ID starting with f0f518fa4ff68de6a8fd17fc6c8e60bff8745c593d4c0669319e72ec6fdaeeaa not found: ID does not exist" Oct 30 18:44:22.527204 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.527147 4603 scope.go:122] "RemoveContainer" containerID="5cb353540add350c5f2dbf1340485e8b8652709eb34c434a65ed2f2a1fb277ba" Oct 30 18:44:22.534641 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.534621 4603 scope.go:122] "RemoveContainer" containerID="5cb353540add350c5f2dbf1340485e8b8652709eb34c434a65ed2f2a1fb277ba" Oct 30 18:44:22.534886 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:44:22.534871 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb353540add350c5f2dbf1340485e8b8652709eb34c434a65ed2f2a1fb277ba\": container with ID starting with 5cb353540add350c5f2dbf1340485e8b8652709eb34c434a65ed2f2a1fb277ba not found: ID does not exist" containerID="5cb353540add350c5f2dbf1340485e8b8652709eb34c434a65ed2f2a1fb277ba" Oct 30 18:44:22.534936 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.534893 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb353540add350c5f2dbf1340485e8b8652709eb34c434a65ed2f2a1fb277ba"} err="failed to get container status \"5cb353540add350c5f2dbf1340485e8b8652709eb34c434a65ed2f2a1fb277ba\": rpc error: code = NotFound desc = could not find container \"5cb353540add350c5f2dbf1340485e8b8652709eb34c434a65ed2f2a1fb277ba\": container with ID starting with 5cb353540add350c5f2dbf1340485e8b8652709eb34c434a65ed2f2a1fb277ba not found: ID does not exist" Oct 30 18:44:22.572663 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.572641 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc"] Oct 30 18:44:22.576914 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.576892 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/distinct-2-w1-b424a5-kserve-55f45b6f97-tq2tc"] Oct 30 18:44:22.641679 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.641656 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf"] Oct 30 18:44:22.646132 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.646110 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/distinct-w1-b424a5-kserve-78c94b7886-6kqxf"] Oct 30 18:44:22.678348 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.678296 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv"] Oct 30 18:44:22.683029 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:22.683010 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/simulator-w1-b424a5-kserve-7d56bc98c8-56hfv"] Oct 30 18:44:23.408738 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:23.408703 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" path="/var/lib/kubelet/pods/bb34381d-3c03-4aa1-9af1-7d000b687af0/volumes" Oct 30 18:44:23.409133 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:23.409080 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" path="/var/lib/kubelet/pods/e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9/volumes" Oct 30 18:44:23.409436 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:23.409423 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" path="/var/lib/kubelet/pods/ea5f82db-b4a7-4c43-8131-dcffe2baa65b/volumes" Oct 30 18:44:30.485618 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:30.485587 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:30.947426 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:30.947393 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:31.099366 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:31.099329 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:31.654186 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:31.654151 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:33.757339 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:33.757302 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-n4z9t"] Oct 30 18:44:37.532719 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:37.532686 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:37.678204 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:37.678166 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:37.857384 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:37.857312 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp"] Oct 30 18:44:37.857567 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:37.857534 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" podUID="e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" containerName="istio-proxy" containerID="cri-o://6d9ee8906105d776e54d58e53db8f078fd097dc052ea82e85fb374224a3319ea" gracePeriod=30 Oct 30 18:44:37.876273 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:37.876234 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q"] Oct 30 18:44:37.876871 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:37.876800 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" podUID="81f49829-73ed-4760-9444-f1793861eba6" containerName="istio-proxy" containerID="cri-o://9c432aa616a5818288b00b8aef7e0e4736137daa708271abf88196d24d5811a8" gracePeriod=30 Oct 30 18:44:37.880001 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:37.879977 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj"] Oct 30 18:44:37.880324 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:37.880292 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" podUID="571fc9dd-71e3-4fa7-b1af-e74f0a96654d" containerName="istio-proxy" containerID="cri-o://310cea5c6e5f288464e3d1d1cbb53ab27bab8bc0ed216f3f3761df8f8bc91bc1" gracePeriod=30 Oct 30 18:44:37.888288 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:37.888262 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr"] Oct 30 18:44:37.888446 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:37.888426 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" podUID="4d6cd466-9f15-476d-9f04-6d03176f025e" containerName="istio-proxy" containerID="cri-o://eb27aae561bd07172ee2db263bba09e608090627f817261fe086fcbe24d81326" gracePeriod=30 Oct 30 18:44:37.927071 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:37.927036 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:39.078133 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:39.078094 4603 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:44:39.078626 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:39.078156 4603 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" podUID="e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:44:42.973679 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:42.973635 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:43.131385 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.131349 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:43.153046 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.153014 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:44:43.211840 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.211817 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:44:43.217884 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.217862 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:44:43.221137 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.221115 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:44:43.316241 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316211 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-workload-certs\") pod \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " Oct 30 18:44:43.316424 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316252 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d6cd466-9f15-476d-9f04-6d03176f025e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d6cd466-9f15-476d-9f04-6d03176f025e-istiod-ca-cert\") pod \"4d6cd466-9f15-476d-9f04-6d03176f025e\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " Oct 30 18:44:43.316424 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316283 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f49829-73ed-4760-9444-f1793861eba6-kube-api-access-lstdn\" (UniqueName: \"kubernetes.io/projected/81f49829-73ed-4760-9444-f1793861eba6-kube-api-access-lstdn\") pod \"81f49829-73ed-4760-9444-f1793861eba6\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " Oct 30 18:44:43.316424 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316307 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-workload-socket\") pod \"4d6cd466-9f15-476d-9f04-6d03176f025e\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " Oct 30 18:44:43.316424 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316334 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-token\" (UniqueName: \"kubernetes.io/projected/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-token\") pod \"4d6cd466-9f15-476d-9f04-6d03176f025e\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " Oct 30 18:44:43.316424 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316366 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-istio-envoy\") pod \"81f49829-73ed-4760-9444-f1793861eba6\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " Oct 30 18:44:43.316424 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316402 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-workload-socket\") pod \"81f49829-73ed-4760-9444-f1793861eba6\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " Oct 30 18:44:43.316801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316430 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-credential-socket\") pod \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " Oct 30 18:44:43.316801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316460 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-data\") pod \"4d6cd466-9f15-476d-9f04-6d03176f025e\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " Oct 30 18:44:43.316801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316496 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-envoy\") pod \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " Oct 30 18:44:43.316801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316602 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-podinfo\") pod \"4d6cd466-9f15-476d-9f04-6d03176f025e\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " Oct 30 18:44:43.316801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316638 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81f49829-73ed-4760-9444-f1793861eba6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f49829-73ed-4760-9444-f1793861eba6-istio-podinfo\") pod \"81f49829-73ed-4760-9444-f1793861eba6\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " Oct 30 18:44:43.316801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316694 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istiod-ca-cert\") pod \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " Oct 30 18:44:43.316801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316727 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-workload-certs\") pod \"4d6cd466-9f15-476d-9f04-6d03176f025e\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " Oct 30 18:44:43.316801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316765 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-workload-certs\") pod \"81f49829-73ed-4760-9444-f1793861eba6\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " Oct 30 18:44:43.316801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316796 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-workload-socket\") pod \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " Oct 30 18:44:43.317292 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316828 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-data\") pod \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " Oct 30 18:44:43.317292 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316880 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-workload-socket\") pod \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " Oct 30 18:44:43.317292 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316927 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-podinfo\") pod \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " Oct 30 18:44:43.317292 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316970 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81f49829-73ed-4760-9444-f1793861eba6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f49829-73ed-4760-9444-f1793861eba6-istiod-ca-cert\") pod \"81f49829-73ed-4760-9444-f1793861eba6\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " Oct 30 18:44:43.317292 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.317005 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-token\" (UniqueName: \"kubernetes.io/projected/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-token\") pod \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " Oct 30 18:44:43.317292 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.317041 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-kube-api-access-ms98v\" (UniqueName: \"kubernetes.io/projected/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-kube-api-access-ms98v\") pod \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " Oct 30 18:44:43.317292 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.317085 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-credential-socket\") pod \"4d6cd466-9f15-476d-9f04-6d03176f025e\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " Oct 30 18:44:43.317292 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.317122 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-token\" (UniqueName: \"kubernetes.io/projected/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-token\") pod \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " Oct 30 18:44:43.317292 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316562 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-workload-certs" pod "571fc9dd-71e3-4fa7-b1af-e74f0a96654d" (UID: "571fc9dd-71e3-4fa7-b1af-e74f0a96654d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.317292 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316720 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d6cd466-9f15-476d-9f04-6d03176f025e-istiod-ca-cert" pod "4d6cd466-9f15-476d-9f04-6d03176f025e" (UID: "4d6cd466-9f15-476d-9f04-6d03176f025e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:43.317292 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316738 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-workload-socket" pod "4d6cd466-9f15-476d-9f04-6d03176f025e" (UID: "4d6cd466-9f15-476d-9f04-6d03176f025e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.317292 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.316825 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-workload-socket" pod "81f49829-73ed-4760-9444-f1793861eba6" (UID: "81f49829-73ed-4760-9444-f1793861eba6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.317292 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.317148 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istiod-ca-cert" pod "571fc9dd-71e3-4fa7-b1af-e74f0a96654d" (UID: "571fc9dd-71e3-4fa7-b1af-e74f0a96654d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:43.318085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.317139 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-credential-socket" pod "571fc9dd-71e3-4fa7-b1af-e74f0a96654d" (UID: "571fc9dd-71e3-4fa7-b1af-e74f0a96654d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.318085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.317434 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-data" pod "4d6cd466-9f15-476d-9f04-6d03176f025e" (UID: "4d6cd466-9f15-476d-9f04-6d03176f025e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.318085 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.317730 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-workload-socket" pod "571fc9dd-71e3-4fa7-b1af-e74f0a96654d" (UID: "571fc9dd-71e3-4fa7-b1af-e74f0a96654d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.319579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.317164 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-podinfo\") pod \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " Oct 30 18:44:43.319579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.318533 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-data\") pod \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " Oct 30 18:44:43.319579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.318580 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-workload-certs\") pod \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " Oct 30 18:44:43.319579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.318611 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-credential-socket\") pod \"81f49829-73ed-4760-9444-f1793861eba6\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " Oct 30 18:44:43.319579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.318644 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-workload-socket" pod "e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" (UID: "e8a63a4e-bf67-43a5-98b0-f38ca2cb7685"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.319579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.318652 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-credential-socket" pod "4d6cd466-9f15-476d-9f04-6d03176f025e" (UID: "4d6cd466-9f15-476d-9f04-6d03176f025e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.319579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.318652 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f49829-73ed-4760-9444-f1793861eba6-istio-token\" (UniqueName: \"kubernetes.io/projected/81f49829-73ed-4760-9444-f1793861eba6-istio-token\") pod \"81f49829-73ed-4760-9444-f1793861eba6\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " Oct 30 18:44:43.319579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.318713 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istiod-ca-cert\") pod \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " Oct 30 18:44:43.319579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.318754 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-credential-socket\") pod \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\" (UID: \"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685\") " Oct 30 18:44:43.319579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.318789 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-envoy\") pod \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " Oct 30 18:44:43.319579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.318884 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-workload-certs" pod "e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" (UID: "e8a63a4e-bf67-43a5-98b0-f38ca2cb7685"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.319579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.318948 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-workload-certs" pod "4d6cd466-9f15-476d-9f04-6d03176f025e" (UID: "4d6cd466-9f15-476d-9f04-6d03176f025e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.319579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.318990 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-workload-certs" pod "81f49829-73ed-4760-9444-f1793861eba6" (UID: "81f49829-73ed-4760-9444-f1793861eba6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.319579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.319006 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-data" pod "571fc9dd-71e3-4fa7-b1af-e74f0a96654d" (UID: "571fc9dd-71e3-4fa7-b1af-e74f0a96654d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.319579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.319135 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-data" pod "e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" (UID: "e8a63a4e-bf67-43a5-98b0-f38ca2cb7685"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.319579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.319248 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-envoy\") pod \"4d6cd466-9f15-476d-9f04-6d03176f025e\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " Oct 30 18:44:43.319579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.319299 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-kube-api-access-t7j8g\" (UniqueName: \"kubernetes.io/projected/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-kube-api-access-t7j8g\") pod \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\" (UID: \"571fc9dd-71e3-4fa7-b1af-e74f0a96654d\") " Oct 30 18:44:43.319579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.319344 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d6cd466-9f15-476d-9f04-6d03176f025e-kube-api-access-hz8kq\" (UniqueName: \"kubernetes.io/projected/4d6cd466-9f15-476d-9f04-6d03176f025e-kube-api-access-hz8kq\") pod \"4d6cd466-9f15-476d-9f04-6d03176f025e\" (UID: \"4d6cd466-9f15-476d-9f04-6d03176f025e\") " Oct 30 18:44:43.319579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.319377 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-istio-data\") pod \"81f49829-73ed-4760-9444-f1793861eba6\" (UID: \"81f49829-73ed-4760-9444-f1793861eba6\") " Oct 30 18:44:43.319579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.319397 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istiod-ca-cert" pod "e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" (UID: "e8a63a4e-bf67-43a5-98b0-f38ca2cb7685"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:43.319579 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.319581 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f49829-73ed-4760-9444-f1793861eba6-istiod-ca-cert" pod "81f49829-73ed-4760-9444-f1793861eba6" (UID: "81f49829-73ed-4760-9444-f1793861eba6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:43.321076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.319821 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-credential-socket" pod "e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" (UID: "e8a63a4e-bf67-43a5-98b0-f38ca2cb7685"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.321076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.319913 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.321076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.319937 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f49829-73ed-4760-9444-f1793861eba6-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.321076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.319953 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.321076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.319968 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.321076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.319984 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.321076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.319997 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.321076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.320010 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.321076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.320024 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d6cd466-9f15-476d-9f04-6d03176f025e-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.321076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.320037 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.321076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.320051 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.321076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.320065 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.321076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.320078 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.321076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.320093 4603 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.321076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.320106 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.321076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.320118 4603 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.321076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.320131 4603 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.321076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.320145 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.321076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.320194 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-credential-socket" pod "81f49829-73ed-4760-9444-f1793861eba6" (UID: "81f49829-73ed-4760-9444-f1793861eba6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.321076 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.320235 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-istio-data" pod "81f49829-73ed-4760-9444-f1793861eba6" (UID: "81f49829-73ed-4760-9444-f1793861eba6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.322784 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.322285 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-token" pod "4d6cd466-9f15-476d-9f04-6d03176f025e" (UID: "4d6cd466-9f15-476d-9f04-6d03176f025e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:43.323004 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.322960 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f49829-73ed-4760-9444-f1793861eba6-kube-api-access-lstdn" pod "81f49829-73ed-4760-9444-f1793861eba6" (UID: "81f49829-73ed-4760-9444-f1793861eba6"). InnerVolumeSpecName "kube-api-access-lstdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:43.323004 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.322962 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f49829-73ed-4760-9444-f1793861eba6-istio-token" pod "81f49829-73ed-4760-9444-f1793861eba6" (UID: "81f49829-73ed-4760-9444-f1793861eba6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:43.323614 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.323198 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-kube-api-access-ms98v" pod "e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" (UID: "e8a63a4e-bf67-43a5-98b0-f38ca2cb7685"). InnerVolumeSpecName "kube-api-access-ms98v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:43.323614 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.323252 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-podinfo" pod "4d6cd466-9f15-476d-9f04-6d03176f025e" (UID: "4d6cd466-9f15-476d-9f04-6d03176f025e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:44:43.323614 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.323437 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6cd466-9f15-476d-9f04-6d03176f025e-kube-api-access-hz8kq" pod "4d6cd466-9f15-476d-9f04-6d03176f025e" (UID: "4d6cd466-9f15-476d-9f04-6d03176f025e"). InnerVolumeSpecName "kube-api-access-hz8kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:43.323977 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.323862 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81f49829-73ed-4760-9444-f1793861eba6-istio-podinfo" pod "81f49829-73ed-4760-9444-f1793861eba6" (UID: "81f49829-73ed-4760-9444-f1793861eba6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:44:43.323977 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.323881 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-envoy" pod "571fc9dd-71e3-4fa7-b1af-e74f0a96654d" (UID: "571fc9dd-71e3-4fa7-b1af-e74f0a96654d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.324143 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.324016 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-istio-envoy" pod "81f49829-73ed-4760-9444-f1793861eba6" (UID: "81f49829-73ed-4760-9444-f1793861eba6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.324204 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.324148 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-envoy" pod "e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" (UID: "e8a63a4e-bf67-43a5-98b0-f38ca2cb7685"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.324300 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.324281 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-podinfo" pod "e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" (UID: "e8a63a4e-bf67-43a5-98b0-f38ca2cb7685"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:44:43.324426 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.324404 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-podinfo" pod "571fc9dd-71e3-4fa7-b1af-e74f0a96654d" (UID: "571fc9dd-71e3-4fa7-b1af-e74f0a96654d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:44:43.324426 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.324415 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-kube-api-access-t7j8g" pod "571fc9dd-71e3-4fa7-b1af-e74f0a96654d" (UID: "571fc9dd-71e3-4fa7-b1af-e74f0a96654d"). InnerVolumeSpecName "kube-api-access-t7j8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:43.324979 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.324961 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-envoy" pod "4d6cd466-9f15-476d-9f04-6d03176f025e" (UID: "4d6cd466-9f15-476d-9f04-6d03176f025e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.325041 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.324973 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-token" pod "571fc9dd-71e3-4fa7-b1af-e74f0a96654d" (UID: "571fc9dd-71e3-4fa7-b1af-e74f0a96654d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:43.325188 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.325169 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-token" pod "e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" (UID: "e8a63a4e-bf67-43a5-98b0-f38ca2cb7685"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:43.422475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.420854 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.422475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.420885 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.422475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.420907 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f49829-73ed-4760-9444-f1793861eba6-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.422475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.420925 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.422475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.420939 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.422475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.420953 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms98v\" (UniqueName: \"kubernetes.io/projected/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-kube-api-access-ms98v\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.422475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.420974 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.422475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.420989 4603 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.422475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.421004 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.422475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.421018 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f49829-73ed-4760-9444-f1793861eba6-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.422475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.421037 4603 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.422475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.421050 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.422475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.421064 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.422475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.421078 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7j8g\" (UniqueName: \"kubernetes.io/projected/571fc9dd-71e3-4fa7-b1af-e74f0a96654d-kube-api-access-t7j8g\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.422475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.421098 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz8kq\" (UniqueName: \"kubernetes.io/projected/4d6cd466-9f15-476d-9f04-6d03176f025e-kube-api-access-hz8kq\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.422475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.421112 4603 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.422475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.421149 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lstdn\" (UniqueName: \"kubernetes.io/projected/81f49829-73ed-4760-9444-f1793861eba6-kube-api-access-lstdn\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.422475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.421163 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d6cd466-9f15-476d-9f04-6d03176f025e-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.422475 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.421181 4603 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f49829-73ed-4760-9444-f1793861eba6-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.618214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.618178 4603 generic.go:356] "Generic (PLEG): container finished" podID="571fc9dd-71e3-4fa7-b1af-e74f0a96654d" containerID="310cea5c6e5f288464e3d1d1cbb53ab27bab8bc0ed216f3f3761df8f8bc91bc1" exitCode=0 Oct 30 18:44:43.618382 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.618245 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" Oct 30 18:44:43.618382 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.618258 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" event={"ID":"571fc9dd-71e3-4fa7-b1af-e74f0a96654d","Type":"ContainerDied","Data":"310cea5c6e5f288464e3d1d1cbb53ab27bab8bc0ed216f3f3761df8f8bc91bc1"} Oct 30 18:44:43.618382 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.618333 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj" event={"ID":"571fc9dd-71e3-4fa7-b1af-e74f0a96654d","Type":"ContainerDied","Data":"615a49976fa88e65a8c5bc32a4aa6a060666326464c2f20e9dbc1f49bf24a373"} Oct 30 18:44:43.618382 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.618362 4603 scope.go:122] "RemoveContainer" containerID="310cea5c6e5f288464e3d1d1cbb53ab27bab8bc0ed216f3f3761df8f8bc91bc1" Oct 30 18:44:43.619656 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.619636 4603 generic.go:356] "Generic (PLEG): container finished" podID="4d6cd466-9f15-476d-9f04-6d03176f025e" containerID="eb27aae561bd07172ee2db263bba09e608090627f817261fe086fcbe24d81326" exitCode=0 Oct 30 18:44:43.619778 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.619697 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" Oct 30 18:44:43.619778 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.619718 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" event={"ID":"4d6cd466-9f15-476d-9f04-6d03176f025e","Type":"ContainerDied","Data":"eb27aae561bd07172ee2db263bba09e608090627f817261fe086fcbe24d81326"} Oct 30 18:44:43.620062 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.619779 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr" event={"ID":"4d6cd466-9f15-476d-9f04-6d03176f025e","Type":"ContainerDied","Data":"1da322e0f9cdb7814bf0a103260960be17f1a8e6bdaa5196b757c6308e8042fb"} Oct 30 18:44:43.621214 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.621192 4603 generic.go:356] "Generic (PLEG): container finished" podID="81f49829-73ed-4760-9444-f1793861eba6" containerID="9c432aa616a5818288b00b8aef7e0e4736137daa708271abf88196d24d5811a8" exitCode=0 Oct 30 18:44:43.621356 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.621223 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" event={"ID":"81f49829-73ed-4760-9444-f1793861eba6","Type":"ContainerDied","Data":"9c432aa616a5818288b00b8aef7e0e4736137daa708271abf88196d24d5811a8"} Oct 30 18:44:43.621356 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.621267 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" Oct 30 18:44:43.621484 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.621268 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q" event={"ID":"81f49829-73ed-4760-9444-f1793861eba6","Type":"ContainerDied","Data":"b6c444c8f6219d6c305ec47e6a84b2e4e1fe586222b1df7312fac134e428fc9f"} Oct 30 18:44:43.623091 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.623072 4603 generic.go:356] "Generic (PLEG): container finished" podID="e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" containerID="6d9ee8906105d776e54d58e53db8f078fd097dc052ea82e85fb374224a3319ea" exitCode=0 Oct 30 18:44:43.623221 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.623103 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" event={"ID":"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685","Type":"ContainerDied","Data":"6d9ee8906105d776e54d58e53db8f078fd097dc052ea82e85fb374224a3319ea"} Oct 30 18:44:43.623221 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.623134 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" Oct 30 18:44:43.623377 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.623134 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp" event={"ID":"e8a63a4e-bf67-43a5-98b0-f38ca2cb7685","Type":"ContainerDied","Data":"90b8dd53bd51c2d39b0c3eea01177e89a69fcdd6c2da68bd9c992021cc3cc4ab"} Oct 30 18:44:43.628067 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.628052 4603 scope.go:122] "RemoveContainer" containerID="310cea5c6e5f288464e3d1d1cbb53ab27bab8bc0ed216f3f3761df8f8bc91bc1" Oct 30 18:44:43.628306 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:44:43.628287 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"310cea5c6e5f288464e3d1d1cbb53ab27bab8bc0ed216f3f3761df8f8bc91bc1\": container with ID starting with 310cea5c6e5f288464e3d1d1cbb53ab27bab8bc0ed216f3f3761df8f8bc91bc1 not found: ID does not exist" containerID="310cea5c6e5f288464e3d1d1cbb53ab27bab8bc0ed216f3f3761df8f8bc91bc1" Oct 30 18:44:43.628359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.628317 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"310cea5c6e5f288464e3d1d1cbb53ab27bab8bc0ed216f3f3761df8f8bc91bc1"} err="failed to get container status \"310cea5c6e5f288464e3d1d1cbb53ab27bab8bc0ed216f3f3761df8f8bc91bc1\": rpc error: code = NotFound desc = could not find container \"310cea5c6e5f288464e3d1d1cbb53ab27bab8bc0ed216f3f3761df8f8bc91bc1\": container with ID starting with 310cea5c6e5f288464e3d1d1cbb53ab27bab8bc0ed216f3f3761df8f8bc91bc1 not found: ID does not exist" Oct 30 18:44:43.628359 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.628333 4603 scope.go:122] "RemoveContainer" containerID="eb27aae561bd07172ee2db263bba09e608090627f817261fe086fcbe24d81326" Oct 30 18:44:43.636215 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.636198 4603 scope.go:122] "RemoveContainer" containerID="eb27aae561bd07172ee2db263bba09e608090627f817261fe086fcbe24d81326" Oct 30 18:44:43.636463 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:44:43.636445 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb27aae561bd07172ee2db263bba09e608090627f817261fe086fcbe24d81326\": container with ID starting with eb27aae561bd07172ee2db263bba09e608090627f817261fe086fcbe24d81326 not found: ID does not exist" containerID="eb27aae561bd07172ee2db263bba09e608090627f817261fe086fcbe24d81326" Oct 30 18:44:43.636537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.636471 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb27aae561bd07172ee2db263bba09e608090627f817261fe086fcbe24d81326"} err="failed to get container status \"eb27aae561bd07172ee2db263bba09e608090627f817261fe086fcbe24d81326\": rpc error: code = NotFound desc = could not find container \"eb27aae561bd07172ee2db263bba09e608090627f817261fe086fcbe24d81326\": container with ID starting with eb27aae561bd07172ee2db263bba09e608090627f817261fe086fcbe24d81326 not found: ID does not exist" Oct 30 18:44:43.636537 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.636487 4603 scope.go:122] "RemoveContainer" containerID="9c432aa616a5818288b00b8aef7e0e4736137daa708271abf88196d24d5811a8" Oct 30 18:44:43.644013 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.643991 4603 scope.go:122] "RemoveContainer" containerID="9c432aa616a5818288b00b8aef7e0e4736137daa708271abf88196d24d5811a8" Oct 30 18:44:43.644253 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:44:43.644234 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c432aa616a5818288b00b8aef7e0e4736137daa708271abf88196d24d5811a8\": container with ID starting with 9c432aa616a5818288b00b8aef7e0e4736137daa708271abf88196d24d5811a8 not found: ID does not exist" containerID="9c432aa616a5818288b00b8aef7e0e4736137daa708271abf88196d24d5811a8" Oct 30 18:44:43.644303 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.644260 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c432aa616a5818288b00b8aef7e0e4736137daa708271abf88196d24d5811a8"} err="failed to get container status \"9c432aa616a5818288b00b8aef7e0e4736137daa708271abf88196d24d5811a8\": rpc error: code = NotFound desc = could not find container \"9c432aa616a5818288b00b8aef7e0e4736137daa708271abf88196d24d5811a8\": container with ID starting with 9c432aa616a5818288b00b8aef7e0e4736137daa708271abf88196d24d5811a8 not found: ID does not exist" Oct 30 18:44:43.644303 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.644275 4603 scope.go:122] "RemoveContainer" containerID="6d9ee8906105d776e54d58e53db8f078fd097dc052ea82e85fb374224a3319ea" Oct 30 18:44:43.651671 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.651655 4603 scope.go:122] "RemoveContainer" containerID="53a9ec2a86a85da94835ae4ea315d8040fd15baad1f9e73f353dd19c01b217a3" Oct 30 18:44:43.659531 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.659480 4603 scope.go:122] "RemoveContainer" containerID="6d9ee8906105d776e54d58e53db8f078fd097dc052ea82e85fb374224a3319ea" Oct 30 18:44:43.659845 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:44:43.659821 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d9ee8906105d776e54d58e53db8f078fd097dc052ea82e85fb374224a3319ea\": container with ID starting with 6d9ee8906105d776e54d58e53db8f078fd097dc052ea82e85fb374224a3319ea not found: ID does not exist" containerID="6d9ee8906105d776e54d58e53db8f078fd097dc052ea82e85fb374224a3319ea" Oct 30 18:44:43.659945 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.659851 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9ee8906105d776e54d58e53db8f078fd097dc052ea82e85fb374224a3319ea"} err="failed to get container status \"6d9ee8906105d776e54d58e53db8f078fd097dc052ea82e85fb374224a3319ea\": rpc error: code = NotFound desc = could not find container \"6d9ee8906105d776e54d58e53db8f078fd097dc052ea82e85fb374224a3319ea\": container with ID starting with 6d9ee8906105d776e54d58e53db8f078fd097dc052ea82e85fb374224a3319ea not found: ID does not exist" Oct 30 18:44:43.659945 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.659866 4603 scope.go:122] "RemoveContainer" containerID="53a9ec2a86a85da94835ae4ea315d8040fd15baad1f9e73f353dd19c01b217a3" Oct 30 18:44:43.660164 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:44:43.660140 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53a9ec2a86a85da94835ae4ea315d8040fd15baad1f9e73f353dd19c01b217a3\": container with ID starting with 53a9ec2a86a85da94835ae4ea315d8040fd15baad1f9e73f353dd19c01b217a3 not found: ID does not exist" containerID="53a9ec2a86a85da94835ae4ea315d8040fd15baad1f9e73f353dd19c01b217a3" Oct 30 18:44:43.660236 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.660172 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53a9ec2a86a85da94835ae4ea315d8040fd15baad1f9e73f353dd19c01b217a3"} err="failed to get container status \"53a9ec2a86a85da94835ae4ea315d8040fd15baad1f9e73f353dd19c01b217a3\": rpc error: code = NotFound desc = could not find container \"53a9ec2a86a85da94835ae4ea315d8040fd15baad1f9e73f353dd19c01b217a3\": container with ID starting with 53a9ec2a86a85da94835ae4ea315d8040fd15baad1f9e73f353dd19c01b217a3 not found: ID does not exist" Oct 30 18:44:43.668968 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.668942 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q"] Oct 30 18:44:43.672987 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.672961 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5th7q"] Oct 30 18:44:43.706546 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.706524 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj"] Oct 30 18:44:43.712632 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.712609 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-chlbj"] Oct 30 18:44:43.745947 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.745925 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp"] Oct 30 18:44:43.754435 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.754414 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-9sdtp"] Oct 30 18:44:43.787789 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.787758 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr"] Oct 30 18:44:43.792243 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:43.792220 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v9skr"] Oct 30 18:44:45.321667 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:45.321628 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:45.408957 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:45.408930 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d6cd466-9f15-476d-9f04-6d03176f025e" path="/var/lib/kubelet/pods/4d6cd466-9f15-476d-9f04-6d03176f025e/volumes" Oct 30 18:44:45.409355 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:45.409341 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="571fc9dd-71e3-4fa7-b1af-e74f0a96654d" path="/var/lib/kubelet/pods/571fc9dd-71e3-4fa7-b1af-e74f0a96654d/volumes" Oct 30 18:44:45.409748 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:45.409734 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81f49829-73ed-4760-9444-f1793861eba6" path="/var/lib/kubelet/pods/81f49829-73ed-4760-9444-f1793861eba6/volumes" Oct 30 18:44:45.410446 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:45.410430 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" path="/var/lib/kubelet/pods/e8a63a4e-bf67-43a5-98b0-f38ca2cb7685/volumes" Oct 30 18:44:50.017130 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:50.017093 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-7bbc6bb9d7-qfgcb_2b9d8afd-1ece-4b82-a62d-b47d6cf5e10b/kuadrant-console-plugin/0.log" Oct 30 18:44:50.531872 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:50.531842 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-f7d6b5c7c-n4z9t_257e1e70-9c25-4ac1-b2e6-c2b2e2a8154d/limitador/0.log" Oct 30 18:44:51.124388 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:44:51.124352 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:51.929348 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:44:51.929313 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:45:19.277115 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:45:19.277078 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:21.929272 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:45:21.929228 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:45:48.001815 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:45:48.001722 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-dd6797bcb-qgcnl_c3c2b36e-0a6b-418d-a91c-b86d4678cb71/maas-api/0.log" Oct 30 18:45:48.622591 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:45:48.622558 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d20c55c3-mxp4f_93bc70e7-e90f-4b6e-886b-9bbbb126d578/revoke-keys/0.log" Oct 30 18:45:51.929720 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:45:51.929685 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:46:01.245850 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:01.245817 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-7bbc6bb9d7-qfgcb_2b9d8afd-1ece-4b82-a62d-b47d6cf5e10b/kuadrant-console-plugin/0.log" Oct 30 18:46:01.804613 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:01.804584 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-f7d6b5c7c-n4z9t_257e1e70-9c25-4ac1-b2e6-c2b2e2a8154d/limitador/0.log" Oct 30 18:46:03.373584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:03.373554 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9mddx_bbab3a56-2437-46f2-ad50-412736486724/discovery/0.log" Oct 30 18:46:03.816383 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:03.816354 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c55b8cc4-7pfgm_e8ba7b34-3707-4350-b3c0-983d6afc6fd3/kube-auth-proxy/0.log" Oct 30 18:46:04.410718 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:04.410692 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-wjstn_c77c24c9-98c1-43c8-9460-e5b3c509c6d3/istio-proxy/0.log" Oct 30 18:46:06.218681 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:06.218646 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-qhvxv_f4343775-794f-4a3c-8732-01eeff138bbe/main/0.log" Oct 30 18:46:06.644169 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:06.644139 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-zz7vx_1112c14a-18f4-46de-ba0c-8e28f19ef80d/main/0.log" Oct 30 18:46:07.049740 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:07.049706 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-7mmmn_f0702ace-9c54-4015-add0-e25ee9feaf40/main/0.log" Oct 30 18:46:11.329397 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.329365 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx"] Oct 30 18:46:11.329810 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.329582 4603 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" podUID="bbab3a56-2437-46f2-ad50-412736486724" containerName="discovery" containerID="cri-o://01af62d12a42db981979a44b783dbe75b6112844973ad418b19204e4493099a2" gracePeriod=30 Oct 30 18:46:11.574994 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.574970 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 18:46:11.620618 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.620563 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbab3a56-2437-46f2-ad50-412736486724-kube-api-access-nh2cn\" (UniqueName: \"kubernetes.io/projected/bbab3a56-2437-46f2-ad50-412736486724-kube-api-access-nh2cn\") pod \"bbab3a56-2437-46f2-ad50-412736486724\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " Oct 30 18:46:11.620749 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.620646 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbab3a56-2437-46f2-ad50-412736486724-istio-token\" (UniqueName: \"kubernetes.io/projected/bbab3a56-2437-46f2-ad50-412736486724-istio-token\") pod \"bbab3a56-2437-46f2-ad50-412736486724\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " Oct 30 18:46:11.620749 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.620696 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbab3a56-2437-46f2-ad50-412736486724-cacerts\" (UniqueName: \"kubernetes.io/secret/bbab3a56-2437-46f2-ad50-412736486724-cacerts\") pod \"bbab3a56-2437-46f2-ad50-412736486724\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " Oct 30 18:46:11.620878 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.620753 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbab3a56-2437-46f2-ad50-412736486724-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bbab3a56-2437-46f2-ad50-412736486724-istio-csr-dns-cert\") pod \"bbab3a56-2437-46f2-ad50-412736486724\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " Oct 30 18:46:11.620878 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.620772 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbab3a56-2437-46f2-ad50-412736486724-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbab3a56-2437-46f2-ad50-412736486724-local-certs\") pod \"bbab3a56-2437-46f2-ad50-412736486724\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " Oct 30 18:46:11.620878 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.620799 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbab3a56-2437-46f2-ad50-412736486724-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbab3a56-2437-46f2-ad50-412736486724-istio-kubeconfig\") pod \"bbab3a56-2437-46f2-ad50-412736486724\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " Oct 30 18:46:11.621067 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.620892 4603 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbab3a56-2437-46f2-ad50-412736486724-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bbab3a56-2437-46f2-ad50-412736486724-istio-csr-ca-configmap\") pod \"bbab3a56-2437-46f2-ad50-412736486724\" (UID: \"bbab3a56-2437-46f2-ad50-412736486724\") " Oct 30 18:46:11.621633 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.621581 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbab3a56-2437-46f2-ad50-412736486724-istio-csr-ca-configmap" pod "bbab3a56-2437-46f2-ad50-412736486724" (UID: "bbab3a56-2437-46f2-ad50-412736486724"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:11.623856 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.623814 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbab3a56-2437-46f2-ad50-412736486724-istio-csr-dns-cert" pod "bbab3a56-2437-46f2-ad50-412736486724" (UID: "bbab3a56-2437-46f2-ad50-412736486724"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:11.623965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.623861 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbab3a56-2437-46f2-ad50-412736486724-kube-api-access-nh2cn" pod "bbab3a56-2437-46f2-ad50-412736486724" (UID: "bbab3a56-2437-46f2-ad50-412736486724"). InnerVolumeSpecName "kube-api-access-nh2cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:11.624419 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.624383 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbab3a56-2437-46f2-ad50-412736486724-local-certs" pod "bbab3a56-2437-46f2-ad50-412736486724" (UID: "bbab3a56-2437-46f2-ad50-412736486724"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:46:11.624662 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.624633 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbab3a56-2437-46f2-ad50-412736486724-cacerts" pod "bbab3a56-2437-46f2-ad50-412736486724" (UID: "bbab3a56-2437-46f2-ad50-412736486724"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:11.625025 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.625005 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbab3a56-2437-46f2-ad50-412736486724-istio-kubeconfig" pod "bbab3a56-2437-46f2-ad50-412736486724" (UID: "bbab3a56-2437-46f2-ad50-412736486724"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:11.625681 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.625660 4603 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbab3a56-2437-46f2-ad50-412736486724-istio-token" pod "bbab3a56-2437-46f2-ad50-412736486724" (UID: "bbab3a56-2437-46f2-ad50-412736486724"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:11.722303 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.722238 4603 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbab3a56-2437-46f2-ad50-412736486724-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:46:11.722303 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.722302 4603 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bbab3a56-2437-46f2-ad50-412736486724-cacerts\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:46:11.722515 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.722314 4603 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bbab3a56-2437-46f2-ad50-412736486724-istio-csr-dns-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:46:11.722515 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.722322 4603 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbab3a56-2437-46f2-ad50-412736486724-local-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:46:11.722515 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.722333 4603 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbab3a56-2437-46f2-ad50-412736486724-istio-kubeconfig\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:46:11.722515 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.722341 4603 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bbab3a56-2437-46f2-ad50-412736486724-istio-csr-ca-configmap\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:46:11.722515 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:11.722349 4603 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh2cn\" (UniqueName: \"kubernetes.io/projected/bbab3a56-2437-46f2-ad50-412736486724-kube-api-access-nh2cn\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\"" Oct 30 18:46:12.010453 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:12.010376 4603 generic.go:356] "Generic (PLEG): container finished" podID="bbab3a56-2437-46f2-ad50-412736486724" containerID="01af62d12a42db981979a44b783dbe75b6112844973ad418b19204e4493099a2" exitCode=0 Oct 30 18:46:12.010453 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:12.010430 4603 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" Oct 30 18:46:12.010677 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:12.010459 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" event={"ID":"bbab3a56-2437-46f2-ad50-412736486724","Type":"ContainerDied","Data":"01af62d12a42db981979a44b783dbe75b6112844973ad418b19204e4493099a2"} Oct 30 18:46:12.010677 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:12.010532 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx" event={"ID":"bbab3a56-2437-46f2-ad50-412736486724","Type":"ContainerDied","Data":"4d83c4c5823ebda01902bd9d9925cfa3b46935db12dd1f8a168d901cd389babe"} Oct 30 18:46:12.010677 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:12.010555 4603 scope.go:122] "RemoveContainer" containerID="01af62d12a42db981979a44b783dbe75b6112844973ad418b19204e4493099a2" Oct 30 18:46:12.030427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:12.030408 4603 scope.go:122] "RemoveContainer" containerID="01af62d12a42db981979a44b783dbe75b6112844973ad418b19204e4493099a2" Oct 30 18:46:12.030711 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:46:12.030691 4603 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01af62d12a42db981979a44b783dbe75b6112844973ad418b19204e4493099a2\": container with ID starting with 01af62d12a42db981979a44b783dbe75b6112844973ad418b19204e4493099a2 not found: ID does not exist" containerID="01af62d12a42db981979a44b783dbe75b6112844973ad418b19204e4493099a2" Oct 30 18:46:12.030774 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:12.030718 4603 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01af62d12a42db981979a44b783dbe75b6112844973ad418b19204e4493099a2"} err="failed to get container status \"01af62d12a42db981979a44b783dbe75b6112844973ad418b19204e4493099a2\": rpc error: code = NotFound desc = could not find container \"01af62d12a42db981979a44b783dbe75b6112844973ad418b19204e4493099a2\": container with ID starting with 01af62d12a42db981979a44b783dbe75b6112844973ad418b19204e4493099a2 not found: ID does not exist" Oct 30 18:46:12.060254 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:12.060232 4603 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx"] Oct 30 18:46:12.071861 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:12.071838 4603 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mddx"] Oct 30 18:46:13.408115 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:13.408083 4603 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbab3a56-2437-46f2-ad50-412736486724" path="/var/lib/kubelet/pods/bbab3a56-2437-46f2-ad50-412736486724/volumes" Oct 30 18:46:16.596711 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:16.596677 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:16.752820 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:16.752788 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:17.470620 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:17.470590 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-jf2vf_ba9c2eff-9e64-4f3f-ba4c-a6339d0afbce/main/0.log" Oct 30 18:46:19.390818 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:19.390779 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:21.928733 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:46:21.928698 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:46:26.001681 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:26.001630 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:26.168639 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:26.168605 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:28.232955 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.232919 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-46zrb/must-gather-2nhgw"] Oct 30 18:46:28.233381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233214 4603 cpu_manager.go:414] "containerMap: removing container" podUID="81f49829-73ed-4760-9444-f1793861eba6" containerName="istio-proxy" Oct 30 18:46:28.233381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233224 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f49829-73ed-4760-9444-f1793861eba6" containerName="istio-proxy" Oct 30 18:46:28.233381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233236 4603 cpu_manager.go:414] "containerMap: removing container" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" Oct 30 18:46:28.233381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233241 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" Oct 30 18:46:28.233381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233252 4603 cpu_manager.go:414] "containerMap: removing container" podUID="e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" containerName="istio-proxy" Oct 30 18:46:28.233381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233258 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" containerName="istio-proxy" Oct 30 18:46:28.233381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233271 4603 cpu_manager.go:414] "containerMap: removing container" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" Oct 30 18:46:28.233381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233276 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" Oct 30 18:46:28.233381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233286 4603 cpu_manager.go:414] "containerMap: removing container" podUID="bbab3a56-2437-46f2-ad50-412736486724" containerName="discovery" Oct 30 18:46:28.233381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233291 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbab3a56-2437-46f2-ad50-412736486724" containerName="discovery" Oct 30 18:46:28.233381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233299 4603 cpu_manager.go:414] "containerMap: removing container" podUID="4d6cd466-9f15-476d-9f04-6d03176f025e" containerName="istio-proxy" Oct 30 18:46:28.233381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233303 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6cd466-9f15-476d-9f04-6d03176f025e" containerName="istio-proxy" Oct 30 18:46:28.233381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233311 4603 cpu_manager.go:414] "containerMap: removing container" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" Oct 30 18:46:28.233381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233315 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" Oct 30 18:46:28.233381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233321 4603 cpu_manager.go:414] "containerMap: removing container" podUID="571fc9dd-71e3-4fa7-b1af-e74f0a96654d" containerName="istio-proxy" Oct 30 18:46:28.233381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233325 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="571fc9dd-71e3-4fa7-b1af-e74f0a96654d" containerName="istio-proxy" Oct 30 18:46:28.233381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233339 4603 cpu_manager.go:414] "containerMap: removing container" podUID="93bc70e7-e90f-4b6e-886b-9bbbb126d578" containerName="revoke-keys" Oct 30 18:46:28.233381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233343 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93bc70e7-e90f-4b6e-886b-9bbbb126d578" containerName="revoke-keys" Oct 30 18:46:28.233381 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233391 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6cd466-9f15-476d-9f04-6d03176f025e" containerName="istio-proxy" Oct 30 18:46:28.234139 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233401 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb34381d-3c03-4aa1-9af1-7d000b687af0" containerName="main" Oct 30 18:46:28.234139 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233406 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="571fc9dd-71e3-4fa7-b1af-e74f0a96654d" containerName="istio-proxy" Oct 30 18:46:28.234139 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233412 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="93bc70e7-e90f-4b6e-886b-9bbbb126d578" containerName="revoke-keys" Oct 30 18:46:28.234139 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233417 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" containerName="istio-proxy" Oct 30 18:46:28.234139 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233421 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f49829-73ed-4760-9444-f1793861eba6" containerName="istio-proxy" Oct 30 18:46:28.234139 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233427 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3e90a36-bf75-4bfa-9f0c-d75dc709f4d9" containerName="main" Oct 30 18:46:28.234139 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233432 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" containerName="istio-proxy" Oct 30 18:46:28.234139 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233438 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5f82db-b4a7-4c43-8131-dcffe2baa65b" containerName="main" Oct 30 18:46:28.234139 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233443 4603 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbab3a56-2437-46f2-ad50-412736486724" containerName="discovery" Oct 30 18:46:28.234139 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233487 4603 cpu_manager.go:414] "containerMap: removing container" podUID="e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" containerName="istio-proxy" Oct 30 18:46:28.234139 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.233491 4603 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a63a4e-bf67-43a5-98b0-f38ca2cb7685" containerName="istio-proxy" Oct 30 18:46:28.237584 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.237568 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-46zrb/must-gather-2nhgw" Oct 30 18:46:28.252143 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.252125 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-46zrb\"/\"kube-root-ca.crt\"" Oct 30 18:46:28.255828 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.255807 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-46zrb\"/\"openshift-service-ca.crt\"" Oct 30 18:46:28.267710 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.267692 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-46zrb\"/\"default-dockercfg-gkpfc\"" Oct 30 18:46:28.268978 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.268960 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-46zrb/must-gather-2nhgw"] Oct 30 18:46:28.336780 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.336755 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:28.350935 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.350916 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lwz4v/must-gather-7ln8m"] Oct 30 18:46:28.355577 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.355538 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lwz4v/must-gather-7ln8m" Oct 30 18:46:28.359459 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.359437 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cffeafbb-0cb4-44a9-a68d-6062537f45f9-must-gather-output\") pod \"must-gather-2nhgw\" (UID: \"cffeafbb-0cb4-44a9-a68d-6062537f45f9\") " pod="openshift-must-gather-46zrb/must-gather-2nhgw" Oct 30 18:46:28.359596 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.359476 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brbps\" (UniqueName: \"kubernetes.io/projected/cffeafbb-0cb4-44a9-a68d-6062537f45f9-kube-api-access-brbps\") pod \"must-gather-2nhgw\" (UID: \"cffeafbb-0cb4-44a9-a68d-6062537f45f9\") " pod="openshift-must-gather-46zrb/must-gather-2nhgw" Oct 30 18:46:28.372841 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.372819 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lwz4v\"/\"default-dockercfg-9zdsh\"" Oct 30 18:46:28.375835 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.375810 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lwz4v\"/\"kube-root-ca.crt\"" Oct 30 18:46:28.375934 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.375886 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lwz4v\"/\"openshift-service-ca.crt\"" Oct 30 18:46:28.379482 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.379465 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lwz4v/must-gather-7ln8m"] Oct 30 18:46:28.460461 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.460433 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cffeafbb-0cb4-44a9-a68d-6062537f45f9-must-gather-output\") pod \"must-gather-2nhgw\" (UID: \"cffeafbb-0cb4-44a9-a68d-6062537f45f9\") " pod="openshift-must-gather-46zrb/must-gather-2nhgw" Oct 30 18:46:28.460608 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.460480 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brbps\" (UniqueName: \"kubernetes.io/projected/cffeafbb-0cb4-44a9-a68d-6062537f45f9-kube-api-access-brbps\") pod \"must-gather-2nhgw\" (UID: \"cffeafbb-0cb4-44a9-a68d-6062537f45f9\") " pod="openshift-must-gather-46zrb/must-gather-2nhgw" Oct 30 18:46:28.460608 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.460543 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c68b7\" (UniqueName: \"kubernetes.io/projected/95417e68-695d-4431-a2fe-ebb65c68a56d-kube-api-access-c68b7\") pod \"must-gather-7ln8m\" (UID: \"95417e68-695d-4431-a2fe-ebb65c68a56d\") " pod="openshift-must-gather-lwz4v/must-gather-7ln8m" Oct 30 18:46:28.460737 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.460623 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/95417e68-695d-4431-a2fe-ebb65c68a56d-must-gather-output\") pod \"must-gather-7ln8m\" (UID: \"95417e68-695d-4431-a2fe-ebb65c68a56d\") " pod="openshift-must-gather-lwz4v/must-gather-7ln8m" Oct 30 18:46:28.460802 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.460783 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cffeafbb-0cb4-44a9-a68d-6062537f45f9-must-gather-output\") pod \"must-gather-2nhgw\" (UID: \"cffeafbb-0cb4-44a9-a68d-6062537f45f9\") " pod="openshift-must-gather-46zrb/must-gather-2nhgw" Oct 30 18:46:28.471645 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.471622 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8rnfq/must-gather-smvn9"] Oct 30 18:46:28.474965 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.474951 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8rnfq/must-gather-smvn9" Oct 30 18:46:28.487015 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.486973 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brbps\" (UniqueName: \"kubernetes.io/projected/cffeafbb-0cb4-44a9-a68d-6062537f45f9-kube-api-access-brbps\") pod \"must-gather-2nhgw\" (UID: \"cffeafbb-0cb4-44a9-a68d-6062537f45f9\") " pod="openshift-must-gather-46zrb/must-gather-2nhgw" Oct 30 18:46:28.491245 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.491231 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8rnfq\"/\"openshift-service-ca.crt\"" Oct 30 18:46:28.491402 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.491384 4603 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8rnfq\"/\"kube-root-ca.crt\"" Oct 30 18:46:28.493227 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.493213 4603 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8rnfq\"/\"default-dockercfg-2cf8n\"" Oct 30 18:46:28.500314 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.500293 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8rnfq/must-gather-smvn9"] Oct 30 18:46:28.551336 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.551315 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-46zrb/must-gather-2nhgw" Oct 30 18:46:28.561234 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.561214 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqdwk\" (UniqueName: \"kubernetes.io/projected/a58a1520-2c5a-4f40-a33a-ad6d7a6adf2d-kube-api-access-zqdwk\") pod \"must-gather-smvn9\" (UID: \"a58a1520-2c5a-4f40-a33a-ad6d7a6adf2d\") " pod="openshift-must-gather-8rnfq/must-gather-smvn9" Oct 30 18:46:28.561334 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.561270 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a58a1520-2c5a-4f40-a33a-ad6d7a6adf2d-must-gather-output\") pod \"must-gather-smvn9\" (UID: \"a58a1520-2c5a-4f40-a33a-ad6d7a6adf2d\") " pod="openshift-must-gather-8rnfq/must-gather-smvn9" Oct 30 18:46:28.561382 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.561362 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c68b7\" (UniqueName: \"kubernetes.io/projected/95417e68-695d-4431-a2fe-ebb65c68a56d-kube-api-access-c68b7\") pod \"must-gather-7ln8m\" (UID: \"95417e68-695d-4431-a2fe-ebb65c68a56d\") " pod="openshift-must-gather-lwz4v/must-gather-7ln8m" Oct 30 18:46:28.561427 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.561410 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/95417e68-695d-4431-a2fe-ebb65c68a56d-must-gather-output\") pod \"must-gather-7ln8m\" (UID: \"95417e68-695d-4431-a2fe-ebb65c68a56d\") " pod="openshift-must-gather-lwz4v/must-gather-7ln8m" Oct 30 18:46:28.561777 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.561747 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/95417e68-695d-4431-a2fe-ebb65c68a56d-must-gather-output\") pod \"must-gather-7ln8m\" (UID: \"95417e68-695d-4431-a2fe-ebb65c68a56d\") " pod="openshift-must-gather-lwz4v/must-gather-7ln8m" Oct 30 18:46:28.588635 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.588612 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c68b7\" (UniqueName: \"kubernetes.io/projected/95417e68-695d-4431-a2fe-ebb65c68a56d-kube-api-access-c68b7\") pod \"must-gather-7ln8m\" (UID: \"95417e68-695d-4431-a2fe-ebb65c68a56d\") " pod="openshift-must-gather-lwz4v/must-gather-7ln8m" Oct 30 18:46:28.589757 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.589726 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:28.662126 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.662091 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqdwk\" (UniqueName: \"kubernetes.io/projected/a58a1520-2c5a-4f40-a33a-ad6d7a6adf2d-kube-api-access-zqdwk\") pod \"must-gather-smvn9\" (UID: \"a58a1520-2c5a-4f40-a33a-ad6d7a6adf2d\") " pod="openshift-must-gather-8rnfq/must-gather-smvn9" Oct 30 18:46:28.662288 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.662156 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a58a1520-2c5a-4f40-a33a-ad6d7a6adf2d-must-gather-output\") pod \"must-gather-smvn9\" (UID: \"a58a1520-2c5a-4f40-a33a-ad6d7a6adf2d\") " pod="openshift-must-gather-8rnfq/must-gather-smvn9" Oct 30 18:46:28.662448 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.662429 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a58a1520-2c5a-4f40-a33a-ad6d7a6adf2d-must-gather-output\") pod \"must-gather-smvn9\" (UID: \"a58a1520-2c5a-4f40-a33a-ad6d7a6adf2d\") " pod="openshift-must-gather-8rnfq/must-gather-smvn9" Oct 30 18:46:28.670995 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.670970 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lwz4v/must-gather-7ln8m" Oct 30 18:46:28.687140 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.687120 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqdwk\" (UniqueName: \"kubernetes.io/projected/a58a1520-2c5a-4f40-a33a-ad6d7a6adf2d-kube-api-access-zqdwk\") pod \"must-gather-smvn9\" (UID: \"a58a1520-2c5a-4f40-a33a-ad6d7a6adf2d\") " pod="openshift-must-gather-8rnfq/must-gather-smvn9" Oct 30 18:46:28.787518 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.787478 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8rnfq/must-gather-smvn9" Oct 30 18:46:28.866948 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.866921 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-46zrb/must-gather-2nhgw"] Oct 30 18:46:28.870941 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:46:28.870913 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcffeafbb_0cb4_44a9_a68d_6062537f45f9.slice/crio-6d44a2fa384e138367ada81cf5e62a8f0b501c46be191968050bd03a8b7fb221 WatchSource:0}: Error finding container 6d44a2fa384e138367ada81cf5e62a8f0b501c46be191968050bd03a8b7fb221: Status 404 returned error can't find the container with id 6d44a2fa384e138367ada81cf5e62a8f0b501c46be191968050bd03a8b7fb221 Oct 30 18:46:28.872732 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.872712 4603 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:46:28.996167 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:28.996146 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lwz4v/must-gather-7ln8m"] Oct 30 18:46:28.998524 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:46:28.998482 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95417e68_695d_4431_a2fe_ebb65c68a56d.slice/crio-bead5db1efd5239e99c7c4fdaddf4bd0e619727bae92ad0c8534abefe4fa9f63 WatchSource:0}: Error finding container bead5db1efd5239e99c7c4fdaddf4bd0e619727bae92ad0c8534abefe4fa9f63: Status 404 returned error can't find the container with id bead5db1efd5239e99c7c4fdaddf4bd0e619727bae92ad0c8534abefe4fa9f63 Oct 30 18:46:29.083200 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:29.083137 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lwz4v/must-gather-7ln8m" event={"ID":"95417e68-695d-4431-a2fe-ebb65c68a56d","Type":"ContainerStarted","Data":"bead5db1efd5239e99c7c4fdaddf4bd0e619727bae92ad0c8534abefe4fa9f63"} Oct 30 18:46:29.084271 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:29.084250 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-46zrb/must-gather-2nhgw" event={"ID":"cffeafbb-0cb4-44a9-a68d-6062537f45f9","Type":"ContainerStarted","Data":"6d44a2fa384e138367ada81cf5e62a8f0b501c46be191968050bd03a8b7fb221"} Oct 30 18:46:29.098070 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:29.098039 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8rnfq/must-gather-smvn9"] Oct 30 18:46:29.100251 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:46:29.100227 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda58a1520_2c5a_4f40_a33a_ad6d7a6adf2d.slice/crio-0d7804abc5d91af1e26db595dfa05c563bb8f4dcb978357e324fa0b1ed40ddb4 WatchSource:0}: Error finding container 0d7804abc5d91af1e26db595dfa05c563bb8f4dcb978357e324fa0b1ed40ddb4: Status 404 returned error can't find the container with id 0d7804abc5d91af1e26db595dfa05c563bb8f4dcb978357e324fa0b1ed40ddb4 Oct 30 18:46:30.113949 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:30.113911 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8rnfq/must-gather-smvn9" event={"ID":"a58a1520-2c5a-4f40-a33a-ad6d7a6adf2d","Type":"ContainerStarted","Data":"0d7804abc5d91af1e26db595dfa05c563bb8f4dcb978357e324fa0b1ed40ddb4"} Oct 30 18:46:30.126387 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:30.126327 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lwz4v/must-gather-7ln8m" event={"ID":"95417e68-695d-4431-a2fe-ebb65c68a56d","Type":"ContainerStarted","Data":"261b5e55a2aa8d30e5c91e8c2e21f3166edecadfd55a5bad39afc08635280423"} Oct 30 18:46:31.140032 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:31.139988 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lwz4v/must-gather-7ln8m" event={"ID":"95417e68-695d-4431-a2fe-ebb65c68a56d","Type":"ContainerStarted","Data":"0a2803571aaad35746633539f768f53590c5af56e3345626eb0bc72cbea72e66"} Oct 30 18:46:33.836714 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:33.836639 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5drgz_1e6590e8-bfc4-4b19-b579-7180aa6af22a/global-pull-secret-syncer/0.log" Oct 30 18:46:34.486125 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:34.486094 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6xn6r_355e77f7-f944-4955-95c2-bdf4c7ea12b7/connection-checker/0.log" Oct 30 18:46:34.771819 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:34.771792 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" Oct 30 18:46:35.122401 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:35.122325 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-236.ec2.internal_f50f91e38dadbc11b70b476e358bc37f/haproxy/0.log" Oct 30 18:46:42.388527 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:42.383074 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-7bbc6bb9d7-qfgcb_2b9d8afd-1ece-4b82-a62d-b47d6cf5e10b/kuadrant-console-plugin/0.log" Oct 30 18:46:43.533111 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:43.533082 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-f7d6b5c7c-n4z9t_257e1e70-9c25-4ac1-b2e6-c2b2e2a8154d/limitador/0.log" Oct 30 18:46:45.363775 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.363291 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-46zrb/must-gather-2nhgw" event={"ID":"cffeafbb-0cb4-44a9-a68d-6062537f45f9","Type":"ContainerStarted","Data":"c1f51aef5d23b8979e3d18b37ff851372f490a10a03cb4dec17534d3a79150be"} Oct 30 18:46:45.363775 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.363345 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-46zrb/must-gather-2nhgw" event={"ID":"cffeafbb-0cb4-44a9-a68d-6062537f45f9","Type":"ContainerStarted","Data":"6363d1402ef06ca41636e750075a41a2c971002e817f2764843f17c5ec76d5a8"} Oct 30 18:46:45.366560 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.366479 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8rnfq/must-gather-smvn9" event={"ID":"a58a1520-2c5a-4f40-a33a-ad6d7a6adf2d","Type":"ContainerStarted","Data":"dbc90c7f29b7dd0c35f1743a0bf8855d5580bdd8edd1121a2f45057a32782744"} Oct 30 18:46:45.366769 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.366741 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8rnfq/must-gather-smvn9" event={"ID":"a58a1520-2c5a-4f40-a33a-ad6d7a6adf2d","Type":"ContainerStarted","Data":"ce7a9ab43aa4a79d1319c9621e1ba647cc23cbce80cafa19db2e610040069f06"} Oct 30 18:46:45.407127 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.407079 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-46zrb/must-gather-2nhgw" podStartSLOduration=2.043700161 podStartE2EDuration="17.40706135s" totalImagesPullingTime="15.363361189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:45.405442485 +0000 UTC m=+3266.121012359" watchObservedRunningTime="2026-10-30 18:46:45.40706135 +0000 UTC m=+3266.122631224" Oct 30 18:46:45.407489 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.407462 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lwz4v/must-gather-7ln8m" podStartSLOduration=16.765840891 podStartE2EDuration="17.407451592s" totalImagesPullingTime="641.610701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:31.190298372 +0000 UTC m=+3251.905868247" watchObservedRunningTime="2026-10-30 18:46:45.407451592 +0000 UTC m=+3266.123021493" Oct 30 18:46:45.451219 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.451146 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8rnfq/must-gather-smvn9" podStartSLOduration=2.302077517 podStartE2EDuration="17.451127805s" totalImagesPullingTime="15.149050288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:45.448987479 +0000 UTC m=+3266.164557354" watchObservedRunningTime="2026-10-30 18:46:45.451127805 +0000 UTC m=+3266.166697675" Oct 30 18:46:45.507023 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.506985 4603 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz"] Oct 30 18:46:45.513175 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.513146 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz" Oct 30 18:46:45.530763 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.530535 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz"] Oct 30 18:46:45.616581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.615795 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn24c\" (UniqueName: \"kubernetes.io/projected/801808aa-a61b-4488-8f55-f8729ebdc0d5-kube-api-access-bn24c\") pod \"perf-node-gather-daemonset-76jpz\" (UID: \"801808aa-a61b-4488-8f55-f8729ebdc0d5\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz" Oct 30 18:46:45.616581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.616354 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/801808aa-a61b-4488-8f55-f8729ebdc0d5-sys\") pod \"perf-node-gather-daemonset-76jpz\" (UID: \"801808aa-a61b-4488-8f55-f8729ebdc0d5\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz" Oct 30 18:46:45.616581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.616463 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/801808aa-a61b-4488-8f55-f8729ebdc0d5-lib-modules\") pod \"perf-node-gather-daemonset-76jpz\" (UID: \"801808aa-a61b-4488-8f55-f8729ebdc0d5\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz" Oct 30 18:46:45.616581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.616496 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/801808aa-a61b-4488-8f55-f8729ebdc0d5-podres\") pod \"perf-node-gather-daemonset-76jpz\" (UID: \"801808aa-a61b-4488-8f55-f8729ebdc0d5\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz" Oct 30 18:46:45.616581 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.616566 4603 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/801808aa-a61b-4488-8f55-f8729ebdc0d5-proc\") pod \"perf-node-gather-daemonset-76jpz\" (UID: \"801808aa-a61b-4488-8f55-f8729ebdc0d5\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz" Oct 30 18:46:45.717594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.717556 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/801808aa-a61b-4488-8f55-f8729ebdc0d5-podres\") pod \"perf-node-gather-daemonset-76jpz\" (UID: \"801808aa-a61b-4488-8f55-f8729ebdc0d5\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz" Oct 30 18:46:45.717801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.717608 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/801808aa-a61b-4488-8f55-f8729ebdc0d5-proc\") pod \"perf-node-gather-daemonset-76jpz\" (UID: \"801808aa-a61b-4488-8f55-f8729ebdc0d5\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz" Oct 30 18:46:45.717801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.717668 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/801808aa-a61b-4488-8f55-f8729ebdc0d5-podres\") pod \"perf-node-gather-daemonset-76jpz\" (UID: \"801808aa-a61b-4488-8f55-f8729ebdc0d5\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz" Oct 30 18:46:45.717801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.717685 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn24c\" (UniqueName: \"kubernetes.io/projected/801808aa-a61b-4488-8f55-f8729ebdc0d5-kube-api-access-bn24c\") pod \"perf-node-gather-daemonset-76jpz\" (UID: \"801808aa-a61b-4488-8f55-f8729ebdc0d5\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz" Oct 30 18:46:45.717801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.717736 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/801808aa-a61b-4488-8f55-f8729ebdc0d5-proc\") pod \"perf-node-gather-daemonset-76jpz\" (UID: \"801808aa-a61b-4488-8f55-f8729ebdc0d5\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz" Oct 30 18:46:45.717801 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.717795 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/801808aa-a61b-4488-8f55-f8729ebdc0d5-sys\") pod \"perf-node-gather-daemonset-76jpz\" (UID: \"801808aa-a61b-4488-8f55-f8729ebdc0d5\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz" Oct 30 18:46:45.718090 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.717823 4603 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/801808aa-a61b-4488-8f55-f8729ebdc0d5-lib-modules\") pod \"perf-node-gather-daemonset-76jpz\" (UID: \"801808aa-a61b-4488-8f55-f8729ebdc0d5\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz" Oct 30 18:46:45.718090 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.717867 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/801808aa-a61b-4488-8f55-f8729ebdc0d5-sys\") pod \"perf-node-gather-daemonset-76jpz\" (UID: \"801808aa-a61b-4488-8f55-f8729ebdc0d5\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz" Oct 30 18:46:45.718090 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.717926 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/801808aa-a61b-4488-8f55-f8729ebdc0d5-lib-modules\") pod \"perf-node-gather-daemonset-76jpz\" (UID: \"801808aa-a61b-4488-8f55-f8729ebdc0d5\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz" Oct 30 18:46:45.742353 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.742320 4603 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn24c\" (UniqueName: \"kubernetes.io/projected/801808aa-a61b-4488-8f55-f8729ebdc0d5-kube-api-access-bn24c\") pod \"perf-node-gather-daemonset-76jpz\" (UID: \"801808aa-a61b-4488-8f55-f8729ebdc0d5\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz" Oct 30 18:46:45.831927 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:45.831892 4603 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz" Oct 30 18:46:46.138950 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:46.138872 4603 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz"] Oct 30 18:46:46.144304 ip-10-0-128-236 kubenswrapper[4603]: W1030 18:46:46.144271 4603 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod801808aa_a61b_4488_8f55_f8729ebdc0d5.slice/crio-a0b5f4a8afaae6439adef0b5c81bdb83d9a5095f7bb248234ad50fb96fcc051f WatchSource:0}: Error finding container a0b5f4a8afaae6439adef0b5c81bdb83d9a5095f7bb248234ad50fb96fcc051f: Status 404 returned error can't find the container with id a0b5f4a8afaae6439adef0b5c81bdb83d9a5095f7bb248234ad50fb96fcc051f Oct 30 18:46:46.380211 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:46.379954 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz" event={"ID":"801808aa-a61b-4488-8f55-f8729ebdc0d5","Type":"ContainerStarted","Data":"482841168d113eebbce2207f0174e94d7a37519e3ac45d4170a12b4e2c1520be"} Oct 30 18:46:46.380737 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:46.380300 4603 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz" event={"ID":"801808aa-a61b-4488-8f55-f8729ebdc0d5","Type":"ContainerStarted","Data":"a0b5f4a8afaae6439adef0b5c81bdb83d9a5095f7bb248234ad50fb96fcc051f"} Oct 30 18:46:46.424942 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:46.424836 4603 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz" podStartSLOduration=1.424817115 podStartE2EDuration="1.424817115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:46.42286055 +0000 UTC m=+3267.138430423" watchObservedRunningTime="2026-10-30 18:46:46.424817115 +0000 UTC m=+3267.140386990" Oct 30 18:46:46.807803 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:46.807776 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7dc5676d65-7f84v_0a90475d-0a43-43eb-9aeb-42804d607d3b/metrics-server/0.log" Oct 30 18:46:47.060711 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:47.060632 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5rlkk_bbbdea07-4cce-465f-a085-3cd5a0e5e334/node-exporter/0.log" Oct 30 18:46:47.207213 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:47.207179 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5rlkk_bbbdea07-4cce-465f-a085-3cd5a0e5e334/kube-rbac-proxy/0.log" Oct 30 18:46:47.329589 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:47.329496 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5rlkk_bbbdea07-4cce-465f-a085-3cd5a0e5e334/init-textfile/0.log" Oct 30 18:46:47.385981 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:47.385950 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz" Oct 30 18:46:50.496146 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:50.496114 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76db67c9b-cvzv8_b970fffa-a462-4914-bec0-3220c1b90dae/telemeter-client/0.log" Oct 30 18:46:50.615407 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:50.615377 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76db67c9b-cvzv8_b970fffa-a462-4914-bec0-3220c1b90dae/reload/0.log" Oct 30 18:46:50.732588 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:50.732562 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76db67c9b-cvzv8_b970fffa-a462-4914-bec0-3220c1b90dae/kube-rbac-proxy/0.log" Oct 30 18:46:51.935479 ip-10-0-128-236 kubenswrapper[4603]: E1030 18:46:51.935448 4603 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsw4r" containerName="extractor" Oct 30 18:46:53.041962 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:53.041921 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-kkgjk_f17fb4c5-0014-4bef-bd0a-7a4f3fc09cf2/networking-console-plugin/0.log" Oct 30 18:46:53.411024 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:53.410961 4603 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-76jpz" Oct 30 18:46:56.347842 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:46:56.347808 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:00.103272 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:00.103234 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7r5h8_8e818352-df23-46b4-84d9-81fd259200ec/dns/0.log" Oct 30 18:47:00.209064 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:00.209028 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7r5h8_8e818352-df23-46b4-84d9-81fd259200ec/kube-rbac-proxy/0.log" Oct 30 18:47:00.696874 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:00.696844 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pbqhh_3ef04039-ef85-40ef-8f30-a0edd7b402e0/dns-node-resolver/0.log" Oct 30 18:47:02.189959 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:02.189832 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4hx5j_088d4d32-c02b-4da2-9435-c5bc22352079/node-ca/0.log" Oct 30 18:47:03.318105 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:03.318065 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c55b8cc4-7pfgm_e8ba7b34-3707-4350-b3c0-983d6afc6fd3/kube-auth-proxy/0.log" Oct 30 18:47:03.536886 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:03.536857 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-wjstn_c77c24c9-98c1-43c8-9460-e5b3c509c6d3/istio-proxy/0.log" Oct 30 18:47:04.871642 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:04.871608 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c55b8cc4-7pfgm_e8ba7b34-3707-4350-b3c0-983d6afc6fd3/kube-auth-proxy/0.log" Oct 30 18:47:05.069594 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:05.069556 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-wjstn_c77c24c9-98c1-43c8-9460-e5b3c509c6d3/istio-proxy/0.log" Oct 30 18:47:05.199331 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:05.199257 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c55b8cc4-7pfgm_e8ba7b34-3707-4350-b3c0-983d6afc6fd3/kube-auth-proxy/0.log" Oct 30 18:47:05.406171 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:05.406136 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:05.412934 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:05.412910 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-wjstn_c77c24c9-98c1-43c8-9460-e5b3c509c6d3/istio-proxy/0.log" Oct 30 18:47:05.618490 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:05.618455 4603 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgghg_1a4bbbf0-cf89-47b9-81cc-9f1e265c2e1f/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:07.281040 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:07.281010 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c55b8cc4-7pfgm_e8ba7b34-3707-4350-b3c0-983d6afc6fd3/kube-auth-proxy/0.log" Oct 30 18:47:07.645072 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:07.644965 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-wjstn_c77c24c9-98c1-43c8-9460-e5b3c509c6d3/istio-proxy/0.log" Oct 30 18:47:09.959779 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:09.959744 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-7bbc6bb9d7-qfgcb_2b9d8afd-1ece-4b82-a62d-b47d6cf5e10b/kuadrant-console-plugin/0.log" Oct 30 18:47:10.163459 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:10.163423 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-f7d6b5c7c-n4z9t_257e1e70-9c25-4ac1-b2e6-c2b2e2a8154d/limitador/0.log" Oct 30 18:47:10.538833 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:10.538796 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-7bbc6bb9d7-qfgcb_2b9d8afd-1ece-4b82-a62d-b47d6cf5e10b/kuadrant-console-plugin/0.log" Oct 30 18:47:10.762961 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:10.762931 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-f7d6b5c7c-n4z9t_257e1e70-9c25-4ac1-b2e6-c2b2e2a8154d/limitador/0.log" Oct 30 18:47:10.812300 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:10.812220 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c55b8cc4-7pfgm_e8ba7b34-3707-4350-b3c0-983d6afc6fd3/kube-auth-proxy/0.log" Oct 30 18:47:10.826709 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:10.826681 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7txkr_c9602b28-165a-4c17-96fa-9f0e25fd2cf8/serve-healthcheck-canary/0.log" Oct 30 18:47:11.009627 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:11.007778 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-wjstn_c77c24c9-98c1-43c8-9460-e5b3c509c6d3/istio-proxy/0.log" Oct 30 18:47:12.686256 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:12.686228 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nsw4r_72993a50-bf5d-40c5-833b-afff9f86e61d/kube-rbac-proxy/0.log" Oct 30 18:47:12.804090 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:12.804060 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nsw4r_72993a50-bf5d-40c5-833b-afff9f86e61d/exporter/0.log" Oct 30 18:47:12.927062 ip-10-0-128-236 kubenswrapper[4603]: I1030 18:47:12.927028 4603 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nsw4r_72993a50-bf5d-40c5-833b-afff9f86e61d/extractor/0.log"